Skip to main content

Welcome

Welcome to the StreamSets Community

94 Topics
HIMANSHU_SURANA
Fan
HIMANSHU_SURANAFan
asked in  Show us your Pipelines
Consuming MySQL binlog data in JDBC producer

I am trying to build CDC pipeline to migrate MySQL database to MySQL database in different server.Here’s the data collector pipeline I’ve created.  As per the documentation here, JDBC producers should be able to process binlog data. I’ve used Field remover to only use /Data and /Table fields.  When I run the pipeline I am getting error that input record has no data for <schema>.<table> . How can I create a pipeline to consume binlog records?

6
Bikram
Headliner
1 year ago
tamilarasup
Discovered Fame
tamilarasupDiscovered Fame
asked in  Show us your Pipelines
Transformer

In Transformer, i sent data from file(directory) to file(local fs)but the destination will got errorJOBRUNNER_63 - Pipeline status: 'RUN_ERROR', message: 'Operator File_3 failed due to org.apache.spark.sql.AnalysisException, check Driver Logs for further information'RUN_ERROR: Operator File_3 failed due to org.apache.spark.sql.AnalysisException, check Driver Logs for further information (View Stack Trace... ) how to clear the error and I need  help to clear the errorThanks Tamilarasu 

6
tamilarasup
Discovered Fame
1 year ago
Drew Kreiger
Rock star
Drew KreigerSenior Community Builder at StreamSets
posted in Welcome Guide
Do you have 2 Minutes? We need your feedback!

StreamSetters we need your feedback! Do you have 2 mins? We are looking for Community member feedback to help us understand how we're doing and how we can better help you on your journey.   

5
Drew Kreiger
Rock star
2 years ago
swayam
Discovered Fame
swayamDiscovered Fame
posted in  Show us your Pipelines
Convert multiple CSV records into a single JSON and make an HTTP bulk API POST call

Hi,The above pipeline reads the CSV files uploaded into an S3 bucket for new contact file uploads. Based on the new contacts available as a CSV, it converts them into a JSON. But the critical part here is that it merges each batches (each batch size is 1000) into a single JSON of multiple records and finally make a single HTTP POST call to one of our internal system to carry out a CONTACT update call.

5
R
Fan
2 years ago
Drew Kreiger
Rock star
Drew KreigerSenior Community Builder at StreamSets
posted in  Show us your Pipelines
Announcement: Share your Pipeline Contest Winners! 🏆

Thank you to the 24 community members who submitted a pipeline! Our two winners for this quarters Share your Pipeline Contest is @swayam and @Bikram! (Stay on the look out for a DM from me.) Check out Swarm’s example pipeline, Carrying out image comparision using SDC pipeline and Amazon Image Rekognition. Where he compares the images received from facebook and twitter to determine if the 2 different profiles belongs to same customer or not. Bikram example Pipeline is, Read multiple csv from S3,Handle corrupt data,Mask the sensitive data and send Data to on premise and Denodo DB. His pipeline reads multiple .csv files from AWS s3 bucket and handles corrupt data if any in the file. The pipeline handles this while masking the data based on DB env and performs full data loads to the destination DB’s.  Stay on the look out for our next Share your Pipeline Contest! We are looking to use Miroverse for ease of trying new example pipeline via one click. 

5
S
Headliner
2 years ago
D
Fan
dixit.singlaFan
asked in  Show us your Pipelines
MongoDB Atlas Connection seems to have a bug.

I was trying to add a MongoDB Atlas connection and on clicking the test connection button, I am getting an error “[unauthorized] not authorized on local to execute command”. I tested the connection string and credentials in Mongo Compass and there it was working fine. So, I could not understand why the test connection is failing. After many tries, I just thought of saving the connection as it is and try it in the pipeline. Magically when used in the pipeline I was able to fetch the data from MongoDB. Then again when I tried the test connection and was getting the same error.Can anyone help me understand this behavior. 

5
lakshmi_narayanan_t
Discovered Fame
7 months ago
shaik111
Fan
shaik111Fan
asked in  Show us your Pipelines
Issue with TCP Server Origin listening port

Am trying to listen some of the port from tcp server origin but its not giving any results, the same port am when using in linux console which we deploy streamsets server am able to listen the message currently am using Version:StreamSets Data Collector 3.19.1  in log am getting TCP_00 - Cannot bind to port [0.0.0.0/0.0.0.0:******]: java.net.BindException: Address already in usehere am hiding port number for confidential with *Could you please help me how to listen messages from streamsets tcp server origin. Thanks in advance 

5
shaik111
Fan
1 year ago
realmatcha
Opening Band
realmatchaOpening Band
asked in  Show us your Pipelines
Connect SqlServer

Hi, I am new to Streamsets, started using it today.I have a task to create some backup data from sqlserver to hadoop and hive, the jdbc connection has been changed to sqlserver but it still doesn't work, please help me

4
realmatcha
Opening Band
1 year ago
G
Fan
gkognoleFan
asked in  Show us your Pipelines
Unable to ingest data from Azure SQL (CDC) to Azure Data bricks using Stream Sets.

Trying to build data pipeline for Azure SQL Server DB (CDC) as source and Azure Data bricks (Delta tables) as destinationI have referred data pipeline sample fromhttps://github.com/streamsets/pipeline-library/tree/master/datacollector/sample-pipelines/pipelines/SQLServer%20CDC%20to%20Delta%20Lake Getting below error for few records in Schema preview as-well:DELTA_LAKE_34 - Databricks Delta Lake load request failed: 'DELTA_LAKE_32 - Could not copy staged file 'sdc-4a076fce-7a73-45ba-8dd7-29e58848cf23.csv': java.sql.SQLException: [Simba][SparkJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.spark.sql.AnalysisException: Unable to infer schema for CSV. It must be specified manually. Note : On Preview/Draft Run → Pipeline is able to capture changes from Source DB, successfully created files in stage (ADLS container) and created Delta tables at destination but it it fails to ingest re

4
G
Fan
2 years ago
Drew Kreiger
Rock star
Drew KreigerSenior Community Builder at StreamSets
posted in  Show us your Pipelines
Share your Pipeline for a chance to win a $250 Amazon Gift Card! Example here!

Streamsetters!  2 Community members will each earn a $250 Amazon Gift card by submitting the most impressive or creative Pipelines. Please share your pipeline within this category (Show us your Pipeline).  It's easy to submit, Make sure to: Create a new Topic in the Show us your Pipeline category. (Each submission needs to have its own topic thread) Add an image of your sample pipeline. (MAKE SURE NO PERSONAL,CREDENTIALS, OR COMPANY INFO IS SCREEN SHOT OR SHARED) Add an explanation of 2-3 sentences of your pipeline use case and benefits. Lastly, Make it fun! Create something useful, fun, creative, and or wacky.   The closing date of our 1st “Share your Pipelines” is March 31, 2022 Example Entry:By @Mike Carley Here are other sample Pipelines. https://github.com/streamsets/pipeline-libraryHow we will choose the three winners: Winner will be announced on Mar 31, 2022.  The two winners will be graded on overall impressions, creativity, originality, and explantation. Winners will be

4
Drew Kreiger
Rock star
2 years ago
A
Discovered Fame
AnkurDiscovered Fame
asked in  Show us your Pipelines
Error: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

Hi,We need to connect AWS S3 Select using Groovy scripting. For that need to upload jar files. While uploading below Jar files, getting the subject line error. The size of these jars is less than 1MB.joda-time.jarhttpclient.jarhttpcore.jaraws-java-sdk-s3.jarCan you please help me resolving this?Error: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms 

3
Bikram
Headliner
1 year ago
D
Discovered Fame
DolphinDiscovered Fame
asked in  Show us your Pipelines
How to Config to make Pipeline "Stop Event" not run when pipeline failed

Hi team,we have a pipeline, which configured a stop event, it let a sql statement to run once pipeline finish processing data,  however we found, when the pipeline failed due to some reason, this “Stop Event” still run, which is not expected for us. Can you please let me know if some place can be configured, which let the Pipeline "Stop Event" not run when pipeline failed processing data? I actually found if the pipelien “Start Event” failed, the pipeline will not run which is expected, however the “Stop Event” always run even pipeline failed. 

2
D
Discovered Fame
5 months ago
moviepages
Fan
moviepagesFan
asked in  Show us your Pipelines
how to create the pipeline for the xml data where it is taken from sftp

how to create the pipeline for the xml data where it is taken from sftp in the streamsets we have getting only 1 record from sftp in the preview data 

2
roma
Fan
5 months ago
Drew Kreiger
Rock star
Drew KreigerSenior Community Builder at StreamSets
posted in Welcome Guide
Meet: Dash, Our First StreamSets Community Champion 🏅

Please meet our first (Q4 2021) StreamSets Quarterly Community Champion, @Dash! Dash has played a foundational role in building this community, cementing himself a figurehead for data engineering expertise. A previous StreamSets teammate, community builder, technical resource with a wealth of knowledge, fabulous dancer, and explorer–it is a delight to continue to have him within our community. It only makes sense to have our first champion be our StreamSets community leader, advocate, and evangelist. Dash has contributed vital efforts building, assisting, and connecting members within our community. His outreach involves helping members with technical questions, co-hosting the Sources and Destinations podcast, participating in DataOps Summit events, and much more. We can't thank him enough. As of today, Dash sits at the "Headliner" Rank with the creation of 27 Topics,  66 Replies, and solving 40 community member's questions!  Over the last 19+ years, Dash has been hands-on within the s

2
Dash
Headliner
2 years ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in  Show us your Pipelines
JDBC PIPE LINE NOT RUNING,SHOWS ME ERROR LIKE: JDBC_16 - Table '' does not exist or PDB is incorrect. Make sure the correct PDB was specified

I facing a issue on running JDBC pipeline ,it shows the below error ,I tried so many things  from the google but no use anybody can solve . 

2
lakshmi_narayanan_t
Discovered Fame
1 year ago
B
Fan
bozo16Fan
asked in  Show us your Pipelines
"What are pipelines?" box gone.

Doing the tutorial, fragments section (https://academy.streamsets.com/learn/courses/6/streamsets-platform-fundamentals/lessons/172/lab-create-a-new-fragment) I’m invite to import a pipeline, but I don’t have the box “What are pipelines?” with the import link. Maybe I closed the box at some stage, but how doI get it back?Screenshot of my Pipelines 

2
B
Fan
9 days ago
S
Discovered Fame
Santhosh KumarDiscovered Fame
asked in  Show us your Pipelines
what is best of streamsets?

Hello Team, My self Santhos kumari'm Beginner of streamsets so which one is best of the learning in depth of streamsets.

2
S
Discovered Fame
1 year ago
nuwancml
Fan
nuwancmlFan
asked in  Show us your Pipelines
Init query in the JDBC Query Consumer not working for Delete previous data

  I use Streamsets Data Collector to Load data from Stage 2 database tables (JDBC Query Consumer) using a query and Write loaded data to another Stage 2 Database table (JDBC Producer). I use Init Query as below to delete the previous records before loading data. But this does not delete any record from the table. It would be great if someone can help me. 

2
nuwancml
Fan
11 months ago
A
Anonymous
posted in Welcome Guide
Hello Community!

My name is Brenna and I’m the new Developer Evangelist here at StreamSets. For the majority of my career, I’ve been a data engineer tasked with moving data from various places within an organization into a central data warehouse. I’m sure this is something you all can relate to. I’m really excited to enable you to get the most out of your journey with StreamSets. I’ll be here in the community answering questions, as well as participating in webinars and podcasts and supporting this developer community any way that I can. As I am ramping up here at StreamSets, please share any content or best practices you’ve found. I would love to learn from you too.

1
Drew Kreiger
Rock star
2 years ago
Drew Kreiger
Rock star
Drew KreigerSenior Community Builder at StreamSets
posted in Welcome Guide
New to the StreamSets Community? Everything to get you started. 🚀

​New to the StreamSets Community? If so, this is the place. Welcome!  First thing first: We are here for YOU. If you are a first-time StreamSets user, customer, partner or a StreamSets Rock star, we have you covered. The StreamSets community is the go-to place to learn, share, and grow your knowledge with other like-minded professionals. If you want to go quickly, go alone. If you want to go far, go together. - African proverb Take a tour of the general welcome community posts.-Please head over to our community guidelines page to learn:Who we are a company and a community team Unacceptable Behavior/ Guidelines (Do’s and Dont’s) Code Of Conduct Communication Guidelines Reporting Harassment or Other IssuesOther Platform related helpful topics:  5 tips to ask your question the right way Question vs Conversation Meet the Community | Introduce yourself Community Platform Theme, Bages, and Ranks DocumentationLearn how to make the most of StreamSets with user guides and tutorials here StreamS

1
sean
StreamSets Employee
3 years ago
Drew Kreiger
Rock star
Drew KreigerSenior Community Builder at StreamSets
posted in Welcome Guide
Community Platform: Feedback

Hello everyone,Welcome to the new StreamSets Community Platform. This Platform will encompass all of our users, customers, partners, and technical employees to assist with peer to peer questions, conversations, and knowledge base articles.  If you have platform feedback, suggestions, or ideas please leave a reply within this thread.  We are continuing to learn and grown. We hope to see you in the community platform. :)   

1
B
Fan
8 months ago
A
Anonymous
posted in  Show us your Pipelines
What's Your Favorite Dinosaur? - StreamSets Pipeline to Clean Delimited Files

Problem StatementYou’ve been asking everybody you meet the age old question: What is your favorite dinosaur? You’ve been saving their responses in a series of delimited files that have this format.  Tragically, the data you have meticulously gathered is dirty. Before you can execute your master plan to do whatever is that you were planning to do with this information, you must clean your dataset and resave it to a single clean file. The PipelineYou read in the Dino data with the Directory origin.  You remove duplicates on the Dinosaur column using the Record Deduplicator You filter Dinos that have murdered people in the Jurassic Park film series for reasons that are you own using the Stream Selector. Finally, you save your Dino data to one single file on your local system. Your data is clean, filtered and ready for analysis.  If you’re reading this, please comment with your favorite dinosaur. It’s uh...for science. 

1
Drew Kreiger
Rock star
2 years ago
Dash
Headliner
DashSenior Technical Evangelist and Developer Advocate at Snowflake
posted in  Show us your Pipelines
Orchestrate Initial Bulk Load to Change Data Capture

 This pipeline is designed to orchestrate initial bulk load to change data capture workload from on-prem Oracle database to Snowflake Data Cloud. The pipeline takes into consideration the completion status of the initial bulk load job before proceeding to the next (CDC) step and also sends success/failure notifications. The pipeline also takes advantage of the platform’s event framework to automatically stop the pipeline when there’s no more data to be ingested. 

1
P
Opening Band
2 years ago
Rishi
StreamSets Employee
RishiStreamSets Employee
posted in  Show us your Pipelines
CDC pipeline from Mysql to Snowflake

 MySQL binary logs provide a very efficient way to track data changes. It contain events that describe the modifications to data. In short, binary logs contain all the information that is required to trace the global status of the server right from the time it was started. This pipeline is design to propagates INSERT, UPDATE, and DELETE operations from MySQL BinLog to Snowflake.

1
Drew Kreiger
Rock star
2 years ago
dima
StreamSets Employee
dimaStreamSets Employee
posted in  Show us your Pipelines
Analyzing chess games with batch and streaming pipelines

I wanted to have a way to send chess game information from lichess.org (a popular chess server) to Elasticsearch and Snowflake to let me visualize statistics (e.g., how often does World Champion GM Magnus Carlsen lose to GM Daniel Naroditsky?) as well as to generate reports in Snowflake (e.g., when Magnus does lose, which openings does he tend to play?). I ended up accomplishing this with two pipelines: This pipeline is a batch pipeline that ingests data using the lichess REST API and pushes it to a Kafka topic. It uses an endpoint that allows for pulling down all games by username, so I’ve parameterized the username portion and then can use Control Hub jobs to pick which specific users I want to study. This pipeline consumes game data from my Kafka topic, does some basic cleanup of the data (adds a field for the name of the winner rather than the color of the winning pieces and converts timestamps from long to datetime) as well as some basic enrichment (it adds a field that calculates

1
R
Fan
6 months ago

Badge winners

Show all badges
Terms & Conditions