Skip to main content

Welcome

Welcome to the StreamSets Community

99 Topics
A
Anonymous
published in Welcome Guide
Last Day! Wisdom of Crowds End-User Survey - Participate Now

Today is the last day to participate in the 2023 Wisdom of Crowds mid-year research process and receive a complimentary report of study findings and a gift from StreamSets as a thank you. Dresner Advisory Services is inviting all business and IT users to participate in its annual examination of the state of the BI, analytics, and information infrastructure marketplaces focusing on drivers, usage, and products.  The 2023 mid-year survey covers Analytical Data Infrastructure (ADI), as well as Embedded BI, Data Engineering, Workforce Planning and Analysis, and Supply Chain Planning and Analysis. Participants only need to fill in areas that are relevant to their current usage and experience. Users in all roles and throughout all industries are invited to contribute their insight via an online survey. The corresponding Dresner Advisory Services research will be published beginning in 4Q23 and throughout 1Q24. Qualified survey participants will receive complimentary copies of the findings.  

160
A
1 year ago
A
Anonymous
published in Welcome Guide
Write a review & receive a $10 gift cardBlog

Hi StreamSets Community! We want to hear from you and ask if you’d share your experience using StreamSets on Capterra. When you write an honest review of your experience on Capterra, the first 30 qualifying reviewers will receive a $10 gift card as a thank you. Your review will help users of technology products and services consider us as their partner for solving their data integration challenges.We appreciate your honest feedback and thank you for your time.Submit your review now

720
A
1 year ago
A
Anonymous
published in Events & Webinars
New Course: Agile Reporting with StreamSets

We are proud to announce a new, free, and completely self-paced course focused on real world scenarios: Agile Reporting with StreamSets.Learn how to streamline financial reporting through modern, resilient, and repeatable data integration with StreamSets.  Quickly and securely integrate and migrate data from internal and external sources, legacy systems, and files into modern data platforms such as Snowflake to accelerate financial reporting.This self-paced hands-on training course teaches you how to overcome several common challenges experienced by organizations attempting to accelerate reporting capabilities:"hard to access" data in legacy systems data silos and data sprawl heavy reliance on IT and a lack of self-service access moving data from legacy to modern data environments integrating internal and external data sources including filesAudienceThe course is designed for data engineers and anyone who is interested in learning how to use StreamSets to solve reporting challenges.Enr

910
A
1 year ago
Drew Kreiger
Rock star
Drew KreigerSenior Community Builder at StreamSets
posted in Welcome Guide
New to the StreamSets Community? Everything to get you started. 🚀

​New to the StreamSets Community? If so, this is the place. Welcome!  First thing first: We are here for YOU. If you are a first-time StreamSets user, customer, partner or a StreamSets Rock star, we have you covered. The StreamSets community is the go-to place to learn, share, and grow your knowledge with other like-minded professionals. If you want to go quickly, go alone. If you want to go far, go together. - African proverb Take a tour of the general welcome community posts.-Please head over to our community guidelines page to learn:Who we are a company and a community team Unacceptable Behavior/ Guidelines (Do’s and Dont’s) Code Of Conduct Communication Guidelines Reporting Harassment or Other IssuesOther Platform related helpful topics:  5 tips to ask your question the right way Question vs Conversation Meet the Community | Introduce yourself Community Platform Theme, Bages, and Ranks DocumentationLearn how to make the most of StreamSets with user guides and tutorials here StreamS

8421
sean
StreamSets Employee
3 years ago
Drew Kreiger
Rock star
Drew KreigerSenior Community Builder at StreamSets
posted in Welcome Guide
StreamSets Community Guidelines

Welcome to the StreamSets Community! Please read over the following Community Guidelines and Code of Conduct below. Please also view StreamSets work culture Blog and Privacy policy. Who we areThe StreamSets Community Is a global community of data engineers who set out to learn, share, and expand their DataOps skills.  “If you want to go fast, go alone. If you want to go far, go together” – African Proverb GuidelinesThe Do’s and Don’ts Do’s: Share helpful docs/resources, tips & tricks, and useful questions Network with other StreamSets users Help answer questions via the forum  Challenge and push boundaries Be positive, Be kindDon’ts: Share personal information Attack/Call out others Degrade other members based on their knowledge level  Promote a personal brand or company Code Of Conduct The StreamSets Community is dedicated to providing a harassment-free, equitable and inclusive community experience for everyone. We do not tolerate harassment of members in any form. We take this po

2350
Drew Kreiger
Rock star
3 years ago
sujith
Fan
sujithFan
asked in  Show us your Pipelines
HAdoop to Azure Migration

How to read multipipe directory data from Hadoop using single pipeline?The below pipeline i have created but i am not sure how define the multiple directory values in the File tab Hadoop FS. Please help us .  

70
sujith
Fan
12 days ago
L
Fan
lotihig137Fan
asked in  Show us your Pipelines
getting Error: [Local FS 1 - Directory Template] Base directory path could not be created

I am trying to build a pipeline by following the steps mentioned in the streamsets docs- https://docs.streamsets.com/portal/platform-controlhub/controlhub/UserGuide/GettingStarted/Try.html#task_q3r_p2x_k4b but getting an issue while writing to the local fils system.the issue resides in the directory template [Directory Template info/HDFS_output/${YYYY()}-${MM()}-${DD()}-${hh()}-${every(5,mm())}codeHADOOPFS_41 - Base directory path could not be created]Can anyone help? Do we have a set of configuration that needs to be installed before running streamsets?

90
L
Fan
1 month ago
J
Fan
jojodoFan
posted in  Show us your Pipelines
MongoDB Atlas Connection seems to have a bug.

I was trying to add a MongoDB Atlas connection and on clicking the test connection button, I am getting an error “[unauthorized] not authorized on local to execute command”. I tested the connection string and credentials in Mongo Compass and there it was working fine. So, I could not understand why the test connection is failing. After many tries, I just thought of saving the connection as it is and try it in the pipeline. Magically when used in the pipeline I was able to fetch the data from MongoDB. Then again when I tried the test connection and was getting the same error.

120
J
Fan
2 months ago
J
Fan
jojodoFan
posted in  Show us your Pipelines
Oracle CDC client pipeline keep running without processing any records, but in DBeaver same account can get data from V$LOGMNR_CONTENTS

I build a pipeline using Oracle CDC client, it is a very simple pipeline , have attached my exported pipeline. I currently using sysdba access configured in Streamsets and using this account I ran below in DBeaver can get records from  V$LOGMNR_CONTENTS, please refer to attached screenshot "DBeaver_logmnr_screeashot.png"", however the streamsets cdc pipeline keeps running without any input and output, I also attached the sdc.log from server.from the log I can see the pipeline has gotten the timestamp of the starting SCN operation , however it cannot get records from LOGMNR and then insert into destination.  

50
J
Fan
2 months ago
B
Fan
bozo16Fan
asked in  Show us your Pipelines
"What are pipelines?" box gone.

Doing the tutorial, fragments section (https://academy.streamsets.com/learn/courses/6/streamsets-platform-fundamentals/lessons/172/lab-create-a-new-fragment) I’m invite to import a pipeline, but I don’t have the box “What are pipelines?” with the import link. Maybe I closed the box at some stage, but how doI get it back?Screenshot of my Pipelines 

372
B
Fan
3 months ago
P
Fan
PratikSinhaFan
asked in  Show us your Pipelines
Error: [Local FS 1 - Directory Template] Base directory path could not be created (HADOOPFS_41)

I am trying to build a pipeline by following the steps mentioned in the streamsets docs- https://docs.streamsets.com/portal/platform-controlhub/controlhub/UserGuide/GettingStarted/Try.html#task_q3r_p2x_k4b but getting an issue while writing to the local fils system.the issue resides in the directory template [Directory Template info/HDFS_output/${YYYY()}-${MM()}-${DD()}-${hh()}-${every(5,mm())}codeHADOOPFS_41 - Base directory path could not be created]Can anyone help? Do we have a set of configuration that needs to be installed before running streamsets?

180
P
Fan
6 months ago
M
Fan
malikhas15Fan
asked in  Show us your Pipelines
Help Needed: Issues Connecting Streamsets JMS Producer to Solace

I have a Streamsets pipeline where I am using the SQL Server CDC component and the JMS Producer component. My connection to SQL Server is working perfectly, and I am receiving data in my pipeline from SQL Server. However, I am encountering issues when trying to connect to Solace through the JMS connector.Initially, I received errors when attempting to connect to Solace. I then used the following parameters:- JMS Initial Context Factory: `com.solacesystems.jndi.SolJNDIInitialContextFactory`- JNDI Connection Factory: `/jms/cf/default`- JMS Provider URL: `tcp://pub:admin@172.17.0.4:55555`- JMS Destination Name: `cdc/demo/sqlserver`- JMS Destination Type: `Topic`- Solace JMS VPN: `test`The pipeline builds successfully, but when I go to the Solace "Try Me" tab and attempt to subscribe to this topic using the client "pub" I do not receive any messages.Additionally, I noticed something peculiar: when I use a client name that does not exist in Solace, the connection still builds successfully.

220
M
Fan
6 months ago
J
JnaneshwariNew Member
posted in  Show us your Pipelines
OBRUNNER_63 - Pipeline status: 'START_ERROR', Pool exhausted'

Hi Team, I am facing bellow error when i execute my pipeline . as a work around we increased ClassLoaders = 100 Still facing same issues  OBRUNNER_63 - Pipeline status: 'START_ERROR', message: 'java.lang.RuntimeException: Could not get a private ClassLoader for 'streamsets-datacollector-azure-lib', for 'com_streamsets_pipeline_stage_metadata_gen2_DataLakeGen2MetadataDExecutor', active private ClassLoaders='50': java.util.NoSuchElementException: Pool exhausted'

70
J
7 months ago
moviepages
Fan
moviepagesFan
asked in  Show us your Pipelines
how to create the pipeline for the xml data where it is taken from sftp

how to create the pipeline for the xml data where it is taken from sftp in the streamsets we have getting only 1 record from sftp in the preview data 

372
roma
Fan
8 months ago
D
Discovered Fame
DolphinDiscovered Fame
asked in  Show us your Pipelines
Cannot Connect StreamSets with Python SDK, but Can connect via CURL command with same Credential ID and Token

Hi Team,I generate Credential ID and Token via StreamSets UI → Manage → API Credential, and with the generated ID and Token I can run Curl command, returned status is “HTTP/1.1 200 OK”, also return me a json format that show my organization id, email, extra.. However, When I want to connect with below code snippet (using same credential ID and Token), it returns me error of 403. paste code and error below:>>> from streamsets.sdk import ControlHub>>> sch = ControlHub(credential_id='absd_myid', token="abcd_mytoken.") Traceback (most recent call last):  File "<stdin>", line 1, in <module>  File "/opt/module/sfsdcvenv/lib/python3.8/site-packages/streamsets/sdk/sch.py", line 141, in __init__    self.api_client = sch_api.ApiClient(  File "/opt/module/sfsdcvenv/lib/python3.8/site-packages/streamsets/sdk/sch_api.py", line 96, in __init__    raise ValueError('Encountered error while decoding auth token: {}'.format(e))ValueError: Encountered error while decoding aut

501
Bikram
Headliner
8 months ago
D
Discovered Fame
DolphinDiscovered Fame
asked in  Show us your Pipelines
How to Config to make Pipeline "Stop Event" not run when pipeline failed

Hi team,we have a pipeline, which configured a stop event, it let a sql statement to run once pipeline finish processing data,  however we found, when the pipeline failed due to some reason, this “Stop Event” still run, which is not expected for us. Can you please let me know if some place can be configured, which let the Pipeline "Stop Event" not run when pipeline failed processing data? I actually found if the pipelien “Start Event” failed, the pipeline will not run which is expected, however the “Stop Event” always run even pipeline failed. 

562
D
Discovered Fame
8 months ago
vishalsdc2024
vishalsdc2024New Member
asked in  Show us your Pipelines
Stream set pipeline is not starting

Hi , i am facing an issue while starting my sdc service and sdc is not coming up. it is prod and need to be up. can someone please help on this at the earliest. Caused by: com.streamsets.datacollector.store.PipelineStoreException: CONTAINER_0206 - Cannot load details for pipeline 'SupplyCha__eed7a3d5-486c-499b-baa1-6eac92aa198b__averydennison.com': java.io.IOException: File '/apps/sdc/data/pipelines/SupplyCha__eed7a3d5-486c-499b-baa1-6eac92aa198b__averydennison.com/pipeline.json-tmp' exists, '/apps/sdc/data/pipelines/SupplyCha__eed7a3d5-486c-499b-baa1-6eac92aa198b__averydennison.com/pipeline.json-old' should exists we have referred link for solution but not sure what can be done as files are not there on those pipelines.  Thanks & Regards,Vishal Verma     

290
vishalsdc2024
9 months ago
C
ColinNew Member
asked in  Show us your Pipelines
Inquiry Regarding StreamSets Product Versions and Feature Differences

As per my understanding, the Community Edition is available for free. Could you please provide insights into the advantages of the Enterprise Edition over the Community Edition? Additionally, what are the limitations of the Community Edition?As per my understanding, the Community Edition is available for free. Could you please provide insights into the advantages of the Enterprise Edition over the Community Edition? Additionally, what are the limitations of the Community Edition?

160
C
9 months ago
D
Discovered Fame
DolphinDiscovered Fame
asked in  Show us your Pipelines
Oracle CDC client pipeline keep running without processing any records, but in DBeaver same account can get data from V$LOGMNR_CONTENTS

Hi, I build a pipeline using Oracle CDC client, it is a very simple pipeline , have attached my exported pipeline. I currently using sysdba access configured in Streamsets and using this account I ran below in DBeaver can get records from  V$LOGMNR_CONTENTS, please refer to attached screenshot "DBeaver_logmnr_screeashot.png"", however the streamsets cdc pipeline keeps running without any input and output, I also attached the sdc.log from server.from the log I can see the pipeline has gotten the timestamp of the starting SCN operation , however it cannot get records from LOGMNR and then insert into destination.can you let know anything wrong here? 

151
D
Discovered Fame
9 months ago
dima
StreamSets Employee
dimaStreamSets Employee
posted in  Show us your Pipelines
Analyzing chess games with batch and streaming pipelines

I wanted to have a way to send chess game information from lichess.org (a popular chess server) to Elasticsearch and Snowflake to let me visualize statistics (e.g., how often does World Champion GM Magnus Carlsen lose to GM Daniel Naroditsky?) as well as to generate reports in Snowflake (e.g., when Magnus does lose, which openings does he tend to play?). I ended up accomplishing this with two pipelines: This pipeline is a batch pipeline that ingests data using the lichess REST API and pushes it to a Kafka topic. It uses an endpoint that allows for pulling down all games by username, so I’ve parameterized the username portion and then can use Control Hub jobs to pick which specific users I want to study. This pipeline consumes game data from my Kafka topic, does some basic cleanup of the data (adds a field for the name of the winner rather than the color of the winning pieces and converts timestamps from long to datetime) as well as some basic enrichment (it adds a field that calculates

1161
R
Fan
9 months ago
Arjo
Fan
ArjoFan
asked in  Show us your Pipelines
"Can values be inserted into another database table from a temporary table that is created inside a function?"

"Can values be inserted into another database table from a temporary table that is created inside a function?"

101
Bikram
Headliner
9 months ago
Arjo
Fan
ArjoFan
posted in  Show us your Pipelines
"Can values be inserted into another database table from a temporary table that is created inside a function?"

"Can values be inserted into another database table from a temporary table that is created inside a function?"

40
Arjo
Fan
9 months ago
realmatcha
Opening Band
realmatchaOpening Band
asked in  Show us your Pipelines
JDBC ERROR

 Error:  Source :  hello everyone, I want to pull data from the source to the warehouse, in the source there is a "customer_id" column, when the streamsets are run why is there an invalid customer error? in streamsets I also flag based on the "id" column. please help, thank you

151
Bikram
Headliner
10 months ago
D
Discovered Fame
DolphinDiscovered Fame
asked in  Show us your Pipelines
About Oracle CDC Client Configure

Hi team,I am configuring Oracle CDC Client now, our oracle version is 19C, non-CDB oracle.I ensure our oracle instance has enabled Database Archiving Mode, cause the return of statement "select log_mode from v$database" is "ARCHIVELOG". My question is for Step "Enable Supplemental Logging", since I have three "NO" for statement "select supplemental_log_data_min, supplemental_log_data_pk, supplemental_log_data_all from v$database;". Is it mandatory to run this "alter database add supplemental log data;" ?I actually do not want to make all database tables' Supplemental Logging, only want to enable it for some related tables, if this, should I only run like below "alter table <schema name>.<table name> add supplemental log data (all) columns;" , then run "alter system archive log current;" or I need to run below in sequence?Step 1:Enable minimal supplemental logging, run "alter database add supplemental log data;"Step 2:alter table <schema name>.<table name> add su

281
D
Discovered Fame
10 months ago
hrishikeh132609
Fan
hrishikeh132609Fan
asked in  Show us your Pipelines
Transformer for spark (Extract and Load).

Hi,Use case.We are load data from Teradata to Snowflake.as we not doing any transformation just extract the Data and loading in Snowflake.Right now, I am using.1.JDBC query consumer (origin) 2.Snowflake (Destination)Note. In our sources we don’t have any primary key or Unique key So, not able to define offset column because of that not able achieve partition the data.   

100
hrishikeh132609
Fan
11 months ago

Badge winners

Show all badges
Terms & Conditions