Skip to main content

- - Knowledge base
Product Updates
Events

30-Day Free Trial: It’s Never Been Easier To Get Started With StreamSets

A

1 year ago

Home
Community overview
StreamSets Platform
Community Articles and Got a Question?

Community Articles and Got a Question?

Can't find what you're looking for? Ask it here or check out the Community articles

1,035 Topics
2,323 Replies

When you subscribe we will email you when there is a new topic in this category

1035 Topics

srinivasa_nanduruFan

asked in Community Articles and Got a Question?

StreamsSets and external events - subscription

Hi,I would like to know if there is a way to subscribe to an external event and trigger a pipeline/job instance in StreamSets Data Collector or Transformer.The example scenario.DDL Change → Database Trigger → StreamSets receives the event → pipleline/job instance triggers when event occurs.

vivekmakodayFan

asked in Community Articles and Got a Question?

Postgres CDC to snowflake Date conversion error

We are using the below pipeline for CDC between Postgres and Snowflake. Pipeline We are getting String to timestamp conversion error on each column which is of date / datetime type. Do you have ay suggestions on how to handle dates so that it can be pushed as dates to snowflake.

asked in Community Articles and Got a Question?

How to use Mover tool to migrate pipeline from Control Hub 3.x to DataOps Platform?

Hello Team,Could you please provide me steps that how to use Mover tool to migrate dataload/data pipeline from Control Hub 3.0 to DataOps Platform. Thanks @Sanjeev @saikat_chatterjee @Mihir Kale , I can see your posts so tagging you here

asked in Community Articles and Got a Question?

The Hype Around Cactus Jack

In the world of streetwear, few names command as much attention as Cactus Jack. Founded by the multifaceted artist Travis Scott, Cactus Jack has become synonymous with high-quality, limited-edition fashion pieces that fuse music, art, and culture. One of the standout items from this brand is the Cactus Jack hoodie, a must-have for any fashion-forward individual.The Origins of Cactus JackBefore diving into why the Cactus Jack hoodie is a wardrobe essential, it’s important to understand the brand's roots. Travis Scott, known for his chart-topping music and electrifying performances, established Cactus Jack to extend his creative vision beyond music. The brand embodies his unique

asked in Community Articles and Got a Question?

Why You Need a Cactus Jack Hoodie in Your Wardrobe

In the world of streetwear, few names command as much attention as Cactus Jack. Founded by the multifaceted artist Travis Scott, Cactus Jack has become synonymous with high-quality, limited-edition fashion pieces that fuse music, art, and culture. One of the standout items from this brand is the Cactus Jack hoodie, a must-have for any fashion-forward individual.The Origins of Cactus JackBefore diving into why the Cactus Jack hoodie is a wardrobe essential, it’s important to understand the brand's roots. Travis Scott, known for his chart-topping music and electrifying performances, established Cactus Jack to extend his creative vision beyond music. The brand embodies his unique

hamiNew Member

posted in Community Articles and Got a Question?

unable to list of jobs via python script

Hello community out there,A newbie to streamsets world.I am looking for automatic solution means via python script with pipeline, which exports all jobs, all pipelines by calling StreamSets Control hub. so I am in very initial stage of my script just for testing, to connect with Control hub and list of jobs. script: streamsets_connection.pyimport osimport sysfrom streamsets.sdk import ControlHub# Get environment variablesserver_url = os.getenv('SDC_SERVER_URL')credential_id = os.getenv('SDC_CREDENTIAL_ID')token = os.getenv('SDC_TOKEN_ID')# Validate environment variablesif not server_url or not credential_id or not token: print("One or more environment variables are not set.") exit(1)# Connect to ControlHubtry: control_hub = ControlHub(server_url=server_url, credential_id=credential_id, token=token) print("ControlHub client initialized successfully.")except Exception as e: print(f"Failed to initialize ControlHub client: {e}") exit(1)def list_jobs(): try: # F

srinivasa_nanduruFan

asked in Community Articles and Got a Question?

ControlHub On-Prem Support

There is a requirement for an ETL tool on-prem.Wanted to check if Streamsets offers ControlHub on-premise optionThanks !Regards,Srinivasa Nanduru

tejas_surveRoadie

asked in Community Articles and Got a Question?

Building REST API with StreamSets using groovy or any evaluator .

We are trying to building REST API with StreamSets using groovy just for exploration.Anyone having any idea regarding this.

Mihir KaleRoadie

asked in Community Articles and Got a Question?

Adding stage libraries in SDC using SDK

I have installed on prem tarball SDC 5.10.0 engine and have registered it with cloud control hub. I want to automate the task of installing/uninstalling stage libraries present in the engine.Is there a way to add stage libraries using SDK? If yes, could you share the appropriate attributes/methods of the SDC object in the SDK and the code snippet?

asked in Community Articles and Got a Question?

StreamSets Data Collector 3.19.1 unable to insert into MapR-DB (MapR 6.1.0)

StreamSets hit issues when trying to insert into MapR-DB, the error as below:Failed executing stage 'MapRDBJSON_01': java.lang.NoSuchMethodError: com.mapr.fs.MapRClientImpl.OpenClient2(Ljava/lang/String;[J[JLjava/lang/String;ZLcom/mapr/fs/jni/MapRClientInitParams;Ljava/lang/String;Ljava/lang/String;Z)JAnyone has any idea about it? How to resolve it?

asked in Community Articles and Got a Question?

Create an empty (NULL) field of type timestamp

Is there a simpler way to create NULL record attributes of a specific type? We are currently doing it that way for TIMESTAMP:

asked in Community Articles and Got a Question?

Streamset testframework isn't working

Hi, Using this blog (https://www.phdata.io/blog/how-to-install-streamsets-test-framework/) by Priya Janjanam, I’m trying to setup Streamsets test framework on my local Windows machine. My local Windows machine has Docker install. I’ve downgraded my python version from 3.12 to python version 3.6 and I followed the instructions in the blog to install framework library and checking the version. However, when I entering the commands below:-stf build extras I get the following error message2024-05-21 02:12:03 PM [INFO] [streamsets.testframework.cli] Pulling Docker image streamsets/testframework-4.x:latest ...Traceback (most recent call last): File "C:\Users\wongp1\PycharmProjects\Streamset-testing\venv36\lib\site-packages\docker\api\client.py", line 268, in _raise_for_status response.raise_for_status() File "C:\Users\wongp1\PycharmProjects\Streamset-testing\venv36\lib\site-packages\requests\models.py", line 960, in raise_for_status raise HTTPError(http_error_msg, response=self)req

Mihir KaleRoadie

asked in Community Articles and Got a Question?

Pipelines not being exported using movertool

I am exporting pipelines from legacy control hub 3.x cloud using mover tool, but few pipelines are not getting exported and are seen as errors in the manifest file. The error message is: '''NoneType'' object is not subscriptable' .I am able to manually export these pipelines and import into DataOps platform.I am an organization admin on control hub and have all the roles associated with exporting pipelines using mover tool.I have shared below screenshots of error object in manifest file and pipeline that did not get exported.Did anyone face this challenge? and what is the meaning of this error?

asked in Community Articles and Got a Question?

Update Job using REST API

I have a requirement to update job parameter via restAPI. I get the job details via GET function. I modify the content via code. But I am not sure how to pass the modified json content in a variable/file in the restapi post call? I have the need to run the api as automated batch with no manual intervention. I tried below two ways but none worked. I dont see any other viable option to pass modified content in the curl command.var1=`cat /home/script/modified.json`curl -X POST https://XXX.hub.streamsets.com/jobrunner/rest/v1/job/72b8200c-0e3b-426e-b564-bceb17220b1e:8c2c652f-e3d9-11eb-9fb3-b974ac4c3f67 -d '{ @var1}’ -H "Content-Type:application/json" -H "X-Requested-By:curl" -H "X-SS-REST-CALL:true" -H "X-SS-App-Component-Id: $CRED_ID" -H "X-SS-App-Auth-Token: $CRED_TOKEN" -i curl -X POST https://XXX.hub.streamsets.com/jobrunner/rest/v1/job/72b8200c-0e3b-426e-b564-bceb17220b1e:8c2c652f-e3d9-11eb-9fb3-b974ac4c3f67 -d '{ /home/script/modified.json}’ -H "Content-Type:application/json" -H "X-R

asked in Community Articles and Got a Question?

Datacollector 'https://streamsets-dc.somedomain:18636' has pipeline in status 'STARTING_ERROR'; message is 'JDBC_09 - Could not retrieve information regarding the columns for table 'SECTION_SCHEDULES''

We are standing up a new Control Hub 3.75 with Data Collectors (DC) version 3.18 and 5.9.1 to run parallel on the new control hub. We are able to ingest data on DC 3.18 with no issues. However, we are receiving the following error on DC 5.9.1:” Datacollector 'https://streamsets:18636' has pipeline in status 'STARTING_ERROR'; message is 'JDBC_09 - Could not retrieve information regarding the columns for table 'SECTION_SCHEDULES''We have run the DDL Job (Start Up) to populate the schema in AWS S3 with no errors on DC 3.18 and DC 5.9.1. The job fails when running DC 5.9.1 on JenzEx and Colleague Student Information Systems.If there is something that someone has come across that would help in troubleshooting this error, it would be greatly appreciated.

jeffersonverissimoFan

asked in Community Articles and Got a Question?

Problem opening pipeline

I'm having trouble trying to open a pipeline in StreamSets.All software is functional, like settings, etc.But, when trying to open a pipeline to edit, for example, it only shows the message "loading". Log messages do not indicate the likely problem.

asked in Community Articles and Got a Question?

Simple JDBC Query Consumer to Local LS

Hello, I am creating the most simplest pipeline: a query (JDBC Query Consumer) whose output is written to files - Local FS. I have broken the files to 200k records each but this job never stops running and keeps writing file after file of what I assume is duplicate data. I cannot for the life of me figure out what I did wrong. Any insight would be greatly appreciated!I have created complex pipelines in the past, so this is strange.

Meghana ChinnuFan

asked in Community Articles and Got a Question?

HTTP Client missing output records

The HTTP Client in my pipeline is not processing all of the input records that it gets. For eg: Input to HTTP client is 1430 records, but the output records processed in the same client are 1360 records only, with 0 error records. Not sure if I am missing any configuration to be added so that I can balance the I/O records, and send the error records to the error stage.

Karan BhatiaFan

asked in Community Articles and Got a Question?

API request using multiple values for a query parameter in one job

Hello There I am trying to solve a very specific usecase here. I am trying to query a DB using an API and this API is using multiple query parameters. one of the query parameters is going to be ids which are more than 100 in count. The catch is that I can’t pass all 100 ids as an array to the API call because API is not designed to accept array for that parameter. It is going to be kind of looping over those 100 ids one by one and then calling the API with new id as the parameter value in each iteration.Also, these IDs needs to be fetched from a snowflake table and then passed as the parameter to the API call. So I am thinking of having some snowflake or JDBC query consumer as as an origin. And these IDs would increase over the period of time so want to make it as dynamic as possible but that is not priority for now. Having multiple jobs to solve this would lead to 100+ jobs and that would keep increasing which is not a good practice at all. Could someone please suggest the best possi

asked in Community Articles and Got a Question?

Kafka Connection error Http failure response for https TUNNELING_INSTANCE_ID=tunneling-1: 500 OK

Dears,I have configured a local docker image for the Streamsets and kafka images to try a simple Kafka connection, but I´m receiving the following:Http failure response for https://na01.hub.streamsets.com/tunneling/rest/660c2f92-c396-4322-a9ea-cd73758897a1/rest/v1/pipeline/dynamicPreview?TUNNELING_INSTANCE_ID=tunneling-1: 500 OK The kafka image names are:bash-3.2$ docker-compose psNAME IMAGE COMMAND SERVICE CREATED STATUS PORTSkafka wurstmeister/kafka "start-kafka.sh" kafka 27 minutes ago Up 27 minutes 0.0.0.0:9092->9092/tcpzookeeper wurstmeister/zookeeper "/bin/sh -c '/usr/sb…" zookeeper 27 minutes ago Up 27 minutes 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp And I try the Test connection kafka:9092 or localhost:9092 and receive the Http failure response for https://na01.hub.streamsets.com/tunneling

jangala.karthikFan

asked in Community Articles and Got a Question?

Change Runtime Parameter value

I have passed a parameter to my pipeline which has a default value when I start the pipeline. I need to change it's value in from one of processors. Is it possible to change the runtime parameter? If yes, can you please provide the steps to do that.

asked in Community Articles and Got a Question?

SNOWFLAKE_74 - Using EL requires auto-creation error

I’m getting this error when I’m trying to use Snowflake uploader. I manually created the stage and I know I can put files there. I’m on 5.8 data collector. How do I over come this error?

asked in Community Articles and Got a Question?

Running Stored procedure and using the data returned by the SP

The case is simple,I have a Kafka topic to which the data gets pushed, I use the data to execute a stored procedure using the JDBC query node. This is working fine but I need to get the return value from the SP so I can log if the execution is a success or there was any error in input which is returned by the SP itself. The SP has some complex logic that involves multiple tables so it's not really good I think to eliminate the SP. Since I don't see any output in JDBC query executor how do I redo the pipeline so I can store/process the SP outputs?

asked in Community Articles and Got a Question?

Snowflake runnable process batch failed: com.streamsets.pipeline.api.StageException: SNOWFLAKE_00 - Could not perform SQL operation: java.sql.SQLException: Error when adding primary keys:

I have a simple pipeline from oracle multi table into snowflake destination. This error only happens when the destination table is not present but it does create the table with PK. After pipeline ran and I can truncate the table and re-run the pipeline without any issues. Any ideas?

inoFan

asked in Community Articles and Got a Question?

Java Throw Error when I using javascript evaluator in SDC

Hello!I'm currently trying out the free (open source version) of SDC. When I created an empty pipeline and added a cron scheduler and JavaScript Evaluator and tried to run it, I encountered the following error:Pipeline Status: STARTING_ERROR: java.lang.NoClassDefFoundError: Could not initialize class com.streamsets.pipeline.stage.processor.javascript.Java8JavaScriptObjectFactoryI was under the impression that using the JavaScript Evaluator did not require any additional package installations. Is there an additional step that needs to be taken?I need some help. Thank you.

1
2
3
4
5
...
42

Page 2 / 42

Badge winners

Ykyeeywkkwhas earned the badge Eager to help
vishwesh.margasahayamhas earned the badge Product expert
ajinkyahas earned the badge Innovator
Sanjeevhas earned the badge Eager to help
AkshayJadhavhas earned the badge Eager to help

Show all badges

Powered by Gainsight

Terms & Conditions

Sign up

Already have an account? Login

Social Login

or

Username *

E-mail address *

What I do... *

Data Leader Data Architect Data Engineer Data Scientist Other

Company *

Country *

Zip Code *

Marketing Communications

Yes No

Password *

I have read and Agree to the Website Terms of Service and I have read and acknowledged the Privacy Policy.

loginBox.register.email_repeat

Login to the community

No account yet? Create an account

Social Login

or

Username or Email

Password

Remember me

Forgot password?

Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.

Enter your e-mail address

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

OK

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.

OK