- - Knowledge base
Product Updates
Events

30-Day Free Trial: It’s Never Been Easier To Get Started With StreamSets

4 months ago

Home
Community overview
Welcome
Show us your Pipelines

Show us your Pipelines

Share an example of your Pipeline pattern. Upload a Screenshot and detailed description. No wrong answers. Do NOT share personal or credential information.

45 Topics
55 Replies

When you subscribe we will email you when there is a new topic in this category

45 Topics

Recently active

Most replies Most views Newest first

asked in Show us your Pipelines

Init query in the JDBC Query Consumer not working for Delete previous data

I use Streamsets Data Collector to Load data from Stage 2 database tables (JDBC Query Consumer) using a query and Write loaded data to another Stage 2 Database table (JDBC Producer). I use Init Query as below to delete the previous records before loading data. But this does not delete any record from the table. It would be great if someone can help me.

asked in Show us your Pipelines

Error while connecting solace event broker

HiI am trying to connect to Solace event broker using JMS Producer and configured all JMS parameters (user name and password ) but still pipeline throwing below error:RETRY: JMS_00 - Could not create initial context 'com.solacesystems.jndi.SolJNDIInitialContextFactory' with provider URL 'tcps://mr-connection-irj89q7fz0j.messaging.solace.cloud:55443' : javax.naming.NamingException: Username must be specified I have already uploaded all solace lib files in external resources section of engine.Please suggest.

drvynguyencosmeticFan

asked in Show us your Pipelines

drvynguyencosmetic

Với cam kết đem đến sự lựa chọn tốt nhất cho khách hàng, Drvynguyencosmetic mang đến một kho lựa chọn ấn tượng từ các thương hiệu hàng đầu như: SVR: Sự ưa chuộng của các chuyên gia da liễu với sản phẩm chất lượng cao. Hãy truy cập ngay https://drvynguyencosmetic.com/ để tìm hiểu rõ hơn về những thông tin trên nhé. Thông tin liên hệ: SĐT: 0931510129Xem thêm:https://drvynguyencosmetic.com/https://www.facebook.com/drvynguyencosmetichttps://www.instagram.com/drvynguyencosmetichttps://twitter.com/dr_vynguyenhttps://www.youtube.com/@drvynguyen.cosmetic/abouthttps://www.flickr.com/people/drvynguyencosmetic/https://www.pinterest.com/drvynguyencosmetic/https://www.tumblr.com/drvynguyencosmetichttps://myspace.com/drvynguyencosmetic

lakshmi_narayanan_tDiscovered Fame

asked in Show us your Pipelines

how to use Google adress api to validate adress column or records

I want to validate the address column by using google address api how to do that .this is my record : I want to validate the address .how to do any one can help this scenario.

asked in Show us your Pipelines

Issue with TCP Server Origin listening port

Am trying to listen some of the port from tcp server origin but its not giving any results, the same port am when using in linux console which we deploy streamsets server am able to listen the message currently am using Version:StreamSets Data Collector 3.19.1 in log am getting TCP_00 - Cannot bind to port [0.0.0.0/0.0.0.0:******]: java.net.BindException: Address already in usehere am hiding port number for confidential with *Could you please help me how to listen messages from streamsets tcp server origin. Thanks in advance

kanyewesthoodieeFan

asked in Show us your Pipelines

How can I compare the results of training?

Hello, CommunityI'm developing a training session. I would need a PRE-training survey and a POST-training survey for this training. The questions (and responses) on both surveys are the same. Ideally, I'd like to show the audience the changes from both the PRE and POST surveys, demonstrating the training's effectiveness on a single screen or report. Additionally, I'd like to provide more fundamental information (such as the average score, the minimum and maximum, the standard deviation, etc.).Does this even exist as a feature, or does anyone know how to accomplish it? I bought the Professional Plan but need help figuring out how to do this. Or does anyone have any other ideas about how to demonstrate improvement?

realmatchaOpening Band

asked in Show us your Pipelines

Connect SqlServer

Hi, I am new to Streamsets, started using it today.I have a task to create some backup data from sqlserver to hadoop and hive, the jdbc connection has been changed to sqlserver but it still doesn't work, please help me

HIMANSHU_SURANAFan

asked in Show us your Pipelines

Consuming MySQL binlog data in JDBC producer

I am trying to build CDC pipeline to migrate MySQL database to MySQL database in different server.Here’s the data collector pipeline I’ve created. As per the documentation here, JDBC producers should be able to process binlog data. I’ve used Field remover to only use /Data and /Table fields. When I run the pipeline I am getting error that input record has no data for <schema>.<table> . How can I create a pipeline to consume binlog records?

asked in Show us your Pipelines

HTTP Server

Can anyone let me know on how to use the HTTP Server Origin stage to fetch large volume of recordsMy problem statement is, I need to fetch large volume of data from a source using GET call. We can use HTTP Client stage for the same but since the source application doesn't have pagination configured , we will receive all the records at once . Thus i want to use HTTP server stage to achieve the same. Can anyone suggest how to achieve this. Appreciate your help!!Thanks in Advance!!

ajinkyaStreamSets Employee

asked in Show us your Pipelines

Sorting a specific column and writing it to a new table

Use Case:-We have a dataset, in which we have columns as follows:-FIRST_NAME, LAST_NAME, EMAIL, PHONE, GENDER, DEPARTMENT, JOB_TITLE, YEARS_OF_EXPEREIENCE, SALARY. Lets sort the column SALARY in ascending order and write to a new table with just 4 columns, FIRST_NAME, LAST_NAME, YEARS_OF_EXPEREIENCE, SALARY. Pipeline Design:-Snowflake Table (origin) Sort (Processor) Column Remover (Processor) Snowflake Table (Destination) Pipeline Working:-Snowflake Origin will fetch the table and columns and pass the records to Sort processor Sort processor will sort the data based on the configuration and pass it to Column Remover. (SALARY column, Ascending order) Column Remover will keep or remove the columns based on the configuration. Snowflake Table Destination will write the data to a new snowflake table.

AnkurDiscovered Fame

asked in Show us your Pipelines

Error: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

Hi,We need to connect AWS S3 Select using Groovy scripting. For that need to upload jar files. While uploading below Jar files, getting the subject line error. The size of these jars is less than 1MB.joda-time.jarhttpclient.jarhttpcore.jaraws-java-sdk-s3.jarCan you please help me resolving this?Error: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

newkanyemerch32Fan

asked in Show us your Pipelines

btsmerchstore45

To operate on an EMR cluster, I have developed a straightforward transformer pipeline. I used the required configuration information during cluster configuration.I have the following problem:

tamilarasupDiscovered Fame

asked in Show us your Pipelines

Transformer

In Transformer, i sent data from file(directory) to file(local fs)but the destination will got errorJOBRUNNER_63 - Pipeline status: 'RUN_ERROR', message: 'Operator File_3 failed due to org.apache.spark.sql.AnalysisException, check Driver Logs for further information'RUN_ERROR: Operator File_3 failed due to org.apache.spark.sql.AnalysisException, check Driver Logs for further information (View Stack Trace... ) how to clear the error and I need help to clear the errorThanks Tamilarasu

tamilarasupDiscovered Fame

asked in Show us your Pipelines

Databricks Transformer

Hinow i am working in streamsets data-ops transformer platform. i try to connect databricks cluster in streamsets. But i facing some issues. JOBRUNNER_63 - Pipeline status: 'START_ERROR', message: 'Failed to start pipeline, check logs for error. The Transformer Spark application in the Spark cluster might not be able to reach Transformer at ‘http://661f1312d1e9:19630’. If this is not the correct URL, update the transformer.base.http.url property in the Transformer configuration file or define a cluster callback URL for the pipeline and restart Transformer.'START_ERROR: Failed to start pipeline, check logs for error. The Transformer Spark application in the Spark cluster might not be able to reach Transformer at ‘http://661f1312d1e9:19630’. If this is not the correct URL, update the transformer.base.http.url property in the Transformer configuration file or define a cluster callback URL for the pipeline and restart Transformer. how to clear this error? and how to connect the cluster to

tamilarasupDiscovered Fame

asked in Show us your Pipelines

mongodb

Is possible to perform cdc in mongodb atlas using mongodb oplog? The mongodb oplog connection string will be satisfied for MongoDB atlas but i got time out errorplease give some guidance to clear that error.

PradeepStreamSets Employee

posted in Show us your Pipelines

Example pipelines for Delta lake lookup processor

Input data:[ { "product_id": 1, "unit_price": 23.2 }, { "product_id": 4, "unit_price": 21.64 }, { "product_id": 2, "unit_price": 10.87 }]Delta lake lookup table. This is same for all operations performedReturn all matching rows, generating a record for each match: Return first matching row:There won’t be change in the output from above since there are no duplicates in input. Return a count of matching rows:Return true if matches exist, otherwise false:Input data:[ { "product_id": 1, "unit_price": 23.2 }, { "product_id": 2, "unit_price": 21.64 }, { "product_id": 4, "unit_price": 10.87 }, { "product_id": 5, "unit_price": 99.32 }, { "product_id": 4, "unit_price": 20.33 }]Return all matching rows, generating a record for each match:Return first matching row:Return true if matches exist, otherwise false: Return a count of matching rows:

Santhosh KumarDiscovered Fame

asked in Show us your Pipelines

what is best of streamsets?

Hello Team, My self Santhos kumari'm Beginner of streamsets so which one is best of the learning in depth of streamsets.

lakshmi_narayanan_tDiscovered Fame

asked in Show us your Pipelines

JDBC PIPE LINE NOT RUNING,SHOWS ME ERROR LIKE: JDBC_16 - Table '' does not exist or PDB is incorrect. Make sure the correct PDB was specified

I facing a issue on running JDBC pipeline ,it shows the below error ,I tried so many things from the google but no use anybody can solve .

swayamDiscovered Fame

posted in Show us your Pipelines

Convert multiple CSV records into a single JSON and make an HTTP bulk API POST call

Hi,The above pipeline reads the CSV files uploaded into an S3 bucket for new contact file uploads. Based on the new contacts available as a CSV, it converts them into a JSON. But the critical part here is that it merges each batches (each batch size is 1000) into a single JSON of multiple records and finally make a single HTTP POST call to one of our internal system to carry out a CONTACT update call.

DashSenior Technical Evangelist and Developer Advocate at Snowflake

posted in Show us your Pipelines

Orchestrate Initial Bulk Load to Change Data Capture

This pipeline is designed to orchestrate initial bulk load to change data capture workload from on-prem Oracle database to Snowflake Data Cloud. The pipeline takes into consideration the completion status of the initial bulk load job before proceeding to the next (CDC) step and also sends success/failure notifications. The pipeline also takes advantage of the platform’s event framework to automatically stop the pipeline when there’s no more data to be ingested.

asked in Show us your Pipelines

Unable to ingest data from Azure SQL (CDC) to Azure Data bricks using Stream Sets.

Trying to build data pipeline for Azure SQL Server DB (CDC) as source and Azure Data bricks (Delta tables) as destinationI have referred data pipeline sample fromhttps://github.com/streamsets/pipeline-library/tree/master/datacollector/sample-pipelines/pipelines/SQLServer%20CDC%20to%20Delta%20Lake Getting below error for few records in Schema preview as-well:DELTA_LAKE_34 - Databricks Delta Lake load request failed: 'DELTA_LAKE_32 - Could not copy staged file 'sdc-4a076fce-7a73-45ba-8dd7-29e58848cf23.csv': java.sql.SQLException: [Simba][SparkJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.spark.sql.AnalysisException: Unable to infer schema for CSV. It must be specified manually. Note : On Preview/Draft Run → Pipeline is able to capture changes from Source DB, successfully created files in stage (ADLS container) and created Delta tables at destination but it it fails to ingest re

iaakashbansalFan

asked in Show us your Pipelines

HTTP Client Origin - Stop Condition for Pagination

I need to extract data from an API using OAuth2 connection. As per the data, they provide /cursor at the end of each page and that cursor can be used to get the records from next page.In Pagination tab, I used Link with Response field and tried to add Stop Condition with /cursor. but, not able to handle this scenario.Can someone please help. Thanks in Advance!!

Drew KreigerSenior Community Builder at StreamSets

posted in Show us your Pipelines

Announcement: Share your Pipeline Contest Winners! 🏆

Thank you to the 24 community members who submitted a pipeline! Our two winners for this quarters Share your Pipeline Contest is @swayam and @Bikram! (Stay on the look out for a DM from me.) Check out Swarm’s example pipeline, Carrying out image comparision using SDC pipeline and Amazon Image Rekognition. Where he compares the images received from facebook and twitter to determine if the 2 different profiles belongs to same customer or not. Bikram example Pipeline is, Read multiple csv from S3,Handle corrupt data,Mask the sensitive data and send Data to on premise and Denodo DB. His pipeline reads multiple .csv files from AWS s3 bucket and handles corrupt data if any in the file. The pipeline handles this while masking the data based on DB env and performs full data loads to the destination DB’s. Stay on the look out for our next Share your Pipeline Contest! We are looking to use Miroverse for ease of trying new example pipeline via one click.

Drew KreigerSenior Community Builder at StreamSets

posted in Show us your Pipelines

Share your Pipeline for a chance to win a $250 Amazon Gift Card! Example here!

Streamsetters! 2 Community members will each earn a $250 Amazon Gift card by submitting the most impressive or creative Pipelines. Please share your pipeline within this category (Show us your Pipeline). It's easy to submit, Make sure to: Create a new Topic in the Show us your Pipeline category. (Each submission needs to have its own topic thread) Add an image of your sample pipeline. (MAKE SURE NO PERSONAL,CREDENTIALS, OR COMPANY INFO IS SCREEN SHOT OR SHARED) Add an explanation of 2-3 sentences of your pipeline use case and benefits. Lastly, Make it fun! Create something useful, fun, creative, and or wacky. The closing date of our 1st “Share your Pipelines” is March 31, 2022 Example Entry:By @Mike Carley Here are other sample Pipelines. https://github.com/streamsets/pipeline-libraryHow we will choose the three winners: Winner will be announced on Mar 31, 2022. The two winners will be graded on overall impressions, creativity, originality, and explantation. Winners will be

swayamDiscovered Fame

posted in Show us your Pipelines

Handling Duplicate Data

Problem StatementVery often we find duplicate records being sent by customer while exporting data from their DWH and sending to us. To ensure data quality, we need to filter out the duplicate records before they enter into our platform for processing, computation and analysis.SolutionWe have addressed this using 2 separate pipelines.Pipeline-1: Files arriving in S3 gets picked up by this pipeline where for every record we calculate the MD5 Hash and make an entry in REDIS with KEY as “The MD5 hash value of the record” and VALUE as “Filename-RowNumber” of the current record being processed. Pipeline-2: The same file processed by Pipeline-1, again gets picked by Pipeline-2 and for every record it calculates the MD5 Hash and checks it’s value in REDIS to determine which RowNumber we need to process and which one we should drop as it’s a duplicate record.Pipeline-1: Pipeline-1: Calculating Hash and Inserting to RedisAs you may see above:CSV file picked from S3 origin, gets converted to JSO

1
2

Page 1 / 2

Badge winners

Sanjeevhas earned the badge Eager to help
AkshayJadhavhas earned the badge Eager to help
john.durkinhas earned the badge Eager to help
samhas earned the badge Eager to help
saleempothiwalahas earned the badge Eager to help

Show all badges

Powered by Gainsight

Terms & Conditions

Sign up

Already have an account? Login

Social Login

or

Username *

E-mail address *

What I do... *

Data Leader Data Architect Data Engineer Data Scientist Other

Company *

Country *

Zip Code *

Marketing Communications

Yes No

Password *

I have read and Agree to the Website Terms of Service and I have read and acknowledged the Privacy Policy.

loginBox.register.email_repeat

Login to the community

No account yet? Create an account

Social Login

or

Username or Email

Password

Remember me

Forgot password?

Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.

Username or e-mail

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

OK

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.

OK