- - Knowledge base
Product Updates
Events

30-Day Free Trial: It’s Never Been Easier To Get Started With StreamSets

1 year ago

Home
Community overview
StreamSets Platform

StreamSets Platform

Get inspired and gain all the knowledge you need

1203 Topics

Most replies

Most views

Newest first

Jay_66Fan

asked in Community Articles and Got a Question?

Is StreamSets Data Collector Version 3.22.3 supported on RHEL8?

Hi,Is the StreamSets Data Collector Version 3.22.3 supported on RHEL8? Thanks,Jay

3 months ago

jose.valdivialeonFan

asked in Community Articles and Got a Question?

Streamsets Postman Collection

Hi all, Is there a Streamsets Postman collection available? If yes, would someone please share the download link?Thank you,

151

3 months ago

srinivasa_nanduruFan

asked in Community Articles and Got a Question?

Global parameter

I want to create a parameter that I can use across multiple pipelines.For example, I have multiple pipelines that are configured to run on EMR cluster.Instead of providing the same EMR configuration values for all the pipelines, if there is a way for me to create parameters globally that I can refer to, in each of the relevant pipelines.Thanks !

312

3 months ago

DolphinDiscovered Fame

asked in StreamSets Academy

Streamsets API Credential not Work, return HTTP/1.1 403 Forbidden Error

HiI am following the steps generate API Credential, and run the content in the green frame directly, however it return error like below. curl -X GET https://eu01.hub.streamsets.com/security/rest/v1/currentUser -H "Content-Type:application/json" -H "X-Requested-By:curl" -H "X-SS-REST-CALL:true" -H "X-SS-App-Component-Id: $CRED_ID" -H "X-SS-App-Auth-Token: $CRED_TOKEN" -i % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 19 100 19 0 0 22 0 --:--:-- --:--:-- --:--:-- 22HTTP/1.1 403 Forbiddencontent-length: 19content-type: text/plaincontent-security-policy: object-src 'none';script-src 'self' https://cdn.cookielaw.org https://privacyportal.onetrust.com https://geolocation.onetrust.com https://app.intercom.io https://widget.intercom.io https://js.intercomcdn.com https://js.userflow.com https://cdn.userflow.com;style-src 'self' https://fonts.googleapis.com h

232

4 months ago

RTiwariFan

asked in Community Articles and Got a Question?

StreamSet Data Collector engine is not working with VPN

Hi,Am using streamSet free version. So for CDC, have created a new deployment and started the engine using generated installation script.But as soon as I connect my vpn, engine stop working (or not start) but I need to connect to vpn for my CDC pipeline work (to connect to origin i.e my oracle db). Please help to resolve this issue. Also, Is there any other way for streamSet CDC setup as I want my standalone streamSet setup to be work in k8s env.TIA!!

4 months ago

Clément ViFan

asked in Community Articles and Got a Question?

Open source On Prem Control Hub package

Good afternoon, I’m currently using Streamsets Data Collector 3.14 (last open source version of sdc).I see that it is possible to connect sdc to a control hub and there is a lot of advantages to do so. Also there is a documentation for installing an on prem control-hub from an archive :https://docs.streamsets.com/portal/controlhub/latest/onpremhelp/controlhub/UserGuide/Install/InstallingDPM.html#concept_exg_11p_hbbUse the following command to extract the tarball:tar xzvf streamsets-dpm-<version>.tar.gz There was even a tutorial for how to build your own sch on prem from a github repo: Howerver the repo is no longer available :http://github.com/streamsets/domainserver Could it be possible to have again access to the repository or to the archive ? Thank you.

170

4 months ago

paulStreamSets Employee

asked in Community Articles and Got a Question?

Details on SDC Metrics

QuestionHow are the SDC Metrics collected and what do they represent?AnswerThe graphs on the SDC Metrics page in StreamSets Data Collector and the Metrics tab of the Execution Engine page in Control Hub are system-level metrics collected from standard Java core libraries. They should roughly correspond with metrics reported by other system-level tools, such as top and uptime at the command line, as well as external monitoring tools that show system-level metrics. Note that the numbers reported by different tools won’t match each other exactly due to differences in reporting intervals and other factors, but there should be a correlation.

431

4 months ago

DolphinDiscovered Fame

asked in StreamSets Academy

Python SDK Start job with Runtime Paramter

Hi I have pipeline and it has 1 runtime parameter about table name. I create a job on top of this pipeline, so it also has a runtime parameter, when create job from this pipeline, I provide a default parameter.I try to use python SDK to call the job, if I ran like below, the job will run with the default parameter(TEST_TABLE1) defined in pipeline and job, it is expected:job = sch.jobs.get(job_name = "MY_SDC_JOB")sch.start_job(job) However when I run like this, I want to overwrite the default runtime parameter, but it still using the default parameter(TEST_TABLE1) , how to make it run with the given parameter(TEST_TABLE2)?sch.start_job(job, TABLE_NAME=’TEST_TABLE2')

260

4 months ago

JVNaiduFan

asked in StreamSets Academy

Can someone help us on the error which i am getting while creating pipeline?

Organization 5c4e8df2-f1c4-11ee-9efd-39b2ca65500f already has 0 pipelines which is the maximum allowed. Please contact support : PIPELINE_STORE_07

111

4 months ago

srinivasa_nanduruFan

asked in Community Articles and Got a Question?

AWS EMR Configuration

I was able to configure EMR Cluster from my transformer pipeline and start the job. But the job does not finish.It fails with an errorERROR Client: Application diagnostics message: Shutdown hook called before final status was reported.Any ideas ?Thanks !Regards,Srinivasa Nanduru

121

4 months ago

vivansaiFan

asked in Community Articles and Got a Question?

How to Update an CyberArk Account

I am trying to update the CyberArk Account using its API.I am following the doc Update AccountThe HTTP method is PATCH according to the documentation.I am trying to make a simple update on my account, following the example from the cyberark docs.

4 months ago

vivansaiFan

asked in Community Articles and Got a Question?

How to Update an Account in CyberArk

4 months ago

jseverinFan

asked in Community Articles and Got a Question?

Pipeline Status: RUN_ERROR: S3_SPOOLDIR_26 - S3 runner failed. Reason org.apache.http.NoHttpResponseException: The target server failed to respond

We have a multi-thread runner listening for new data on an S3 bucket and processing and filtering the data. Our pipeline seems to work well for a few hours, but gets this error below. We are getting about 5-10MB of data per second and that gets filtered down to about 2% of data through our processing and filtering. Is this due to too much data or a configuration problem? Pipeline Status: RUN_ERROR: S3_SPOOLDIR_26 - S3 runner failed. Reason org.apache.http.NoHttpResponseException: The target server failed to respond

761

5 months ago

sujatha mogiliFan

asked in Community Articles and Got a Question?

The streamsets pipeline fails with java.lang.OutOfMemoryError: Java heap space

The streamsets pipeline fails on a regular basis with the below error. Please advise if you ran into this issue and the resolution. We have increased Heap size couple of times but not helping.“ERROR A JVM error occurred while running the pipeline, java.lang.OutOfMemoryError: Java heap space”

5916

5 months ago

surajpFan

asked in StreamSets Academy

Transformer Engine does not start Automatically

Hi AllI have a trial account of Stream sets cloud.I am going through the streamsets-platform-fundamentals course.So i am currently trying to get hands-on , on the transformer engine.The Transformer Deployment and engines were installed and running yesterday, i also built 2 pipelines there. then i shut down my lab environment provided by StreamSets.When i login today and start the above lab environment, all the data collector engines start automatically, but the Transformer Engine does not start automatically.There is no option to start the transformer engine either. As you can see from the below screenshot. If i select the Restart Engine or Shut Down Engine i get the below errorEngine cannot be reached. Verify that the engine is running. Because of this now i am not able to proceed further with the course, can someone help me on how to start the transformer engine after restarting my Lab Environment ?

322

5 months ago

srinivasa_nanduruFan

asked in Community Articles and Got a Question?

StreamsSets and external events - subscription

Hi,I would like to know if there is a way to subscribe to an external event and trigger a pipeline/job instance in StreamSets Data Collector or Transformer.The example scenario.DDL Change → Database Trigger → StreamSets receives the event → pipleline/job instance triggers when event occurs.

5 months ago

vivekmakodayFan

asked in Community Articles and Got a Question?

Postgres CDC to snowflake Date conversion error

We are using the below pipeline for CDC between Postgres and Snowflake. Pipeline We are getting String to timestamp conversion error on each column which is of date / datetime type. Do you have ay suggestions on how to handle dates so that it can be pushed as dates to snowflake.

130

5 months ago

Dev_WorkFan

asked in Community Articles and Got a Question?

How to use Mover tool to migrate pipeline from Control Hub 3.x to DataOps Platform?

Hello Team,Could you please provide me steps that how to use Mover tool to migrate dataload/data pipeline from Control Hub 3.0 to DataOps Platform. Thanks @Sanjeev @saikat_chatterjee @Mihir Kale , I can see your posts so tagging you here

171

5 months ago

CorteizHoodieFan

asked in Share your Best Practices

Brand Philosophy and Mission

In today's fashion landscape, the hoodie has evolved from a simple piece of sportswear to a versatile wardrobe staple. Among the myriad of brands offering this essential garment, Corteiz stands out for its unique blend of style, comfort, and quality. As a rising star in the US fashion market, Corteiz has garnered a loyal following thanks to its dedication to craftsmanship and innovative designs. This article delves into the world of Corteiz hoodies, exploring everything from their design and material to customer reviews and styling tips. Whether you're new to the brand or a longtime fan, this comprehensive guide will provide all the information you need about Corteiz hoodies.

5 months ago

CorteizHoodieFan

asked in Share your Best Practices

The Corteiz Brand

110

5 months ago

CactusjackFan

asked in Community Articles and Got a Question?

The Hype Around Cactus Jack

In the world of streetwear, few names command as much attention as Cactus Jack. Founded by the multifaceted artist Travis Scott, Cactus Jack has become synonymous with high-quality, limited-edition fashion pieces that fuse music, art, and culture. One of the standout items from this brand is the Cactus Jack hoodie, a must-have for any fashion-forward individual.The Origins of Cactus JackBefore diving into why the Cactus Jack hoodie is a wardrobe essential, it’s important to understand the brand's roots. Travis Scott, known for his chart-topping music and electrifying performances, established Cactus Jack to extend his creative vision beyond music. The brand embodies his unique

190

5 months ago

CactusjackFan

asked in Community Articles and Got a Question?

Why You Need a Cactus Jack Hoodie in Your Wardrobe

5 months ago

hamiNew Member

posted in Community Articles and Got a Question?

unable to list of jobs via python script

Hello community out there,A newbie to streamsets world.I am looking for automatic solution means via python script with pipeline, which exports all jobs, all pipelines by calling StreamSets Control hub. so I am in very initial stage of my script just for testing, to connect with Control hub and list of jobs. script: streamsets_connection.pyimport osimport sysfrom streamsets.sdk import ControlHub# Get environment variablesserver_url = os.getenv('SDC_SERVER_URL')credential_id = os.getenv('SDC_CREDENTIAL_ID')token = os.getenv('SDC_TOKEN_ID')# Validate environment variablesif not server_url or not credential_id or not token: print("One or more environment variables are not set.") exit(1)# Connect to ControlHubtry: control_hub = ControlHub(server_url=server_url, credential_id=credential_id, token=token) print("ControlHub client initialized successfully.")except Exception as e: print(f"Failed to initialize ControlHub client: {e}") exit(1)def list_jobs(): try: # F

501

5 months ago

srinivasa_nanduruFan

asked in Community Articles and Got a Question?

ControlHub On-Prem Support

There is a requirement for an ETL tool on-prem.Wanted to check if Streamsets offers ControlHub on-premise optionThanks !Regards,Srinivasa Nanduru

142

5 months ago

tejas_surveRoadie

asked in Community Articles and Got a Question?

Building REST API with StreamSets using groovy or any evaluator .

We are trying to building REST API with StreamSets using groovy just for exploration.Anyone having any idea regarding this.

131

5 months ago

Page 2 / 49

Badge winners

Ykyeeywkkwhas earned the badge Eager to help
vishwesh.margasahayamhas earned the badge Product expert
ajinkyahas earned the badge Innovator
Sanjeevhas earned the badge Eager to help
AkshayJadhavhas earned the badge Eager to help

Show all badges

Terms & Conditions

Sign up

Already have an account? Login

Social Login

Username *

E-mail address *

What I do... *

Data Leader Data Architect Data Engineer Data Scientist Other

Company *

Country *

Zip Code *

Marketing Communications

Yes No

Password *

I have read and Agree to the Website Terms of Service and I have read and acknowledged the Privacy Policy.

loginBox.register.email_repeat

Login to the community

No account yet? Create an account

Social Login

Username or Email

Password

Remember me

Forgot password?

Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.

Enter your e-mail address

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.