Welcome

 Input data:[  {    "product_id": 1,    "unit_price": 23.2  },  {    "product_id": 4,    "unit_price": 21.64  },  {    "product_id": 2,    "unit_price": 10.87  }]Delta lake lookup table. This is same for all operations performedReturn all matching rows, generating a record for each match:  Return first matching row:There won’t be change in the output from above since there are no duplicates in input. Return a count of matching rows:Return true if matches exist, otherwise false:Input data:[  {    "product_id": 1,    "unit_price": 23.2  },  {    "product_id": 2,    "unit_price": 21.64  },  {    "product_id": 4,    "unit_price": 10.87  },    {    "product_id": 5,    "unit_price": 99.32  },    {    "product_id": 4,    "unit_price": 20.33  }]Return all matching rows, generating a record for each match:Return first matching row:Return true if matches exist, otherwise false: Return a count of matching rows: 

Hello Team, My self Santhos kumari'm Beginner of streamsets so which one is best of the learning in depth of streamsets.

I facing a issue on running JDBC pipeline ,it shows the below error ,I tried so many things  from the google but no use anybody can solve . 

I’m excited to write that we’ve announced dates for the StreamSets Roadshow. (Details here and in your right hand tool bar) First upcoming are San Francisco and New York. I wanted to personally invite all my fellow data engineers to join me at my training sessions during these events.If you use my code BuuckSF50 for the San Francisco Roadshow or BuuckNYC50 for New York you can get 50% off a 4 hour training and a half-day of info on best practices and innovation with StreamSets. I hope to see you there!

Hi,The above pipeline reads the CSV files uploaded into an S3 bucket for new contact file uploads. Based on the new contacts available as a CSV, it converts them into a JSON. But the critical part here is that it merges each batches (each batch size is 1000) into a single JSON of multiple records and finally make a single HTTP POST call to one of our internal system to carry out a CONTACT update call.

 This pipeline is designed to orchestrate initial bulk load to change data capture workload from on-prem Oracle database to Snowflake Data Cloud. The pipeline takes into consideration the completion status of the initial bulk load job before proceeding to the next (CDC) step and also sends success/failure notifications. The pipeline also takes advantage of the platform’s event framework to automatically stop the pipeline when there’s no more data to be ingested. 

Trying to build data pipeline for Azure SQL Server DB (CDC) as source and Azure Data bricks (Delta tables) as destinationI have referred data pipeline sample fromhttps://github.com/streamsets/pipeline-library/tree/master/datacollector/sample-pipelines/pipelines/SQLServer%20CDC%20to%20Delta%20Lake Getting below error for few records in Schema preview as-well:DELTA_LAKE_34 - Databricks Delta Lake load request failed: 'DELTA_LAKE_32 - Could not copy staged file 'sdc-4a076fce-7a73-45ba-8dd7-29e58848cf23.csv': java.sql.SQLException: [Simba][SparkJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.spark.sql.AnalysisException: Unable to infer schema for CSV. It must be specified manually. Note : On Preview/Draft Run → Pipeline is able to capture changes from Source DB, successfully created files in stage (ADLS container) and created Delta tables at destination but it it fails to ingest re

I need to extract data from an API using OAuth2 connection. As per the data, they provide /cursor at the end of each page and that cursor can be used to get the records from next page.In Pagination tab, I used Link with Response field and tried to add Stop Condition with /cursor. but, not able to handle this scenario.Can someone please help. Thanks in Advance!!

We are excited to announce our new offering: StreamSets Transformer for Snowflake! Learn details about the new offering and about how we're shifting to a higher gear. Gain the insights here, https://blog.softwareag.com/streamsets-snowflake

We have launched a new program called Meetsy! A program around bringing each other closer to connect and learn from each other through automated 1:1 meetings. StreamSets technical employees will be joining this program! This is a great way to ask questions and learn from StreamSets experts! Sign up here!  *Make sure to fill out the questioner as the tool better aligns members with similar goals and wants from the program. You may want to join our StreamSetters community Slack for easier notification etc. I hope to see you part of the Meetsy program! Please ask your questions within this thread. 

Today we have crossed 500 community members here in our community platform! 🎉 I personally wanted to say thank you to everyone who has joined, asked questions, helped others, and made this an overall great place where everyone is kindly welcomed. I am excited to continue to build relationships with everyone. Let’s continue learning, sharing, and growing together! Thank you.   -@Drew Kreiger 

Tomorrow! #1day2goJoin Snowpark Day to learn how to make advanced transformations and data processing in Snowflake easy. Design and run simple to complex transformations on your data natively in Snowflake with an intuitive, visual design. Not your time zone, no problem - register for access to on-demand. ❄️ RSVP here

StreamSetters!  We are very pleased to announce that StreamSets has officially joined the Software AG family. Software AG, the software pioneer of a truly connected world, closed its acquisition of StreamSets on April 18. StreamSets and Software AG are committed to continued investment in the StreamSets DataOps Platform and in the success of our Community. We are excited to bring value through new programs and resources.  A few words from our CEO Girish Pancha: “At StreamSets, we believe in harnessing change to drive business transformation. With our DataOps Platform, we ensure data engineering teams thrive in a world of constant change. Now, we are thrilled to announce our own change. StreamSets is becoming an autonomous business within Software AG, which will bring additional resources and new opportunities. But our mission remains the same: delivering the industry-leading data integration platform so our customers can thrive in an evolving, hybrid world.”We look forward to continuin

This detailed, step-by-step guide will introduce you to the StreamSets DataOps Platform architectural concepts and the steps you’ll need to take to build and run your smart data pipelines. https://go.streamsets.com/rs/535-TEA-657/images/StreamSets-Starters-Guide-Architecture.pdf

Thank you to the 24 community members who submitted a pipeline! Our two winners for this quarters Share your Pipeline Contest is @swayam and @Bikram! (Stay on the look out for a DM from me.) Check out Swarm’s example pipeline, Carrying out image comparision using SDC pipeline and Amazon Image Rekognition. Where he compares the images received from facebook and twitter to determine if the 2 different profiles belongs to same customer or not. Bikram example Pipeline is, Read multiple csv from S3,Handle corrupt data,Mask the sensitive data and send Data to on premise and Denodo DB. His pipeline reads multiple .csv files from AWS s3 bucket and handles corrupt data if any in the file. The pipeline handles this while masking the data based on DB env and performs full data loads to the destination DB’s.  Stay on the look out for our next Share your Pipeline Contest! We are looking to use Miroverse for ease of trying new example pipeline via one click. 

Streamsetters!  2 Community members will each earn a $250 Amazon Gift card by submitting the most impressive or creative Pipelines. Please share your pipeline within this category (Show us your Pipeline).  It's easy to submit, Make sure to: Create a new Topic in the Show us your Pipeline category. (Each submission needs to have its own topic thread) Add an image of your sample pipeline. (MAKE SURE NO PERSONAL,CREDENTIALS, OR COMPANY INFO IS SCREEN SHOT OR SHARED) Add an explanation of 2-3 sentences of your pipeline use case and benefits. Lastly, Make it fun! Create something useful, fun, creative, and or wacky.   The closing date of our 1st “Share your Pipelines” is March 31, 2022 Example Entry:By @Mike Carley Here are other sample Pipelines. https://github.com/streamsets/pipeline-libraryHow we will choose the three winners: Winner will be announced on Mar 31, 2022.  The two winners will be graded on overall impressions, creativity, originality, and explantation. Winners will be

Problem StatementVery often we find duplicate records being sent by customer while exporting data from their DWH and sending to us. To ensure data quality, we need to filter out the duplicate records before they enter into our platform for processing, computation and analysis.SolutionWe have addressed this using 2 separate pipelines.Pipeline-1: Files arriving in S3 gets picked up by this pipeline where for every record we calculate the MD5 Hash and make an entry in REDIS with KEY as “The MD5 hash value of the record” and VALUE as “Filename-RowNumber” of the current record being processed. Pipeline-2: The same file processed by Pipeline-1, again gets picked by Pipeline-2 and for every record it calculates the MD5 Hash and checks it’s value in REDIS to determine which RowNumber we need to process and which one we should drop as it’s a duplicate record.Pipeline-1:  Pipeline-1: Calculating Hash and Inserting to RedisAs you may see above:CSV file picked from S3 origin, gets converted to JSO

 Streaming data pipeline that potentially has updates to transactions. Look up for matching records in the database (postgres), filter transaction using a condition to determine if transactions are updates vs new.  For updates, mask sensitive data and writes them to postgres db table.  New transaction are written to an AWS bucket and unprocessed transactions (pipeline errors) are written to a kafka destination for further analysis.

We built this pipeline to compare the images received from facebook and twitter to determine if the 2 different profiles belongs to same customer ordifferent.We have a business case where we pull the social media comments from different users received from facebook and twitter. Apart from carrying out the sentiment analysis, we also need to know if same customer is being vocal in multiple platforms. Since customers normally don’t share their PII data, when we pull comments along with profile images from facebook and twitter, we store the images and consider the image matching as one of the criteria to determine the probability score of two comments received from different platfroms belongs to the same person or not.Before running the above pipeline, we have the pre-processed data with some probability score where the name, city, age, sentiments are already matched to come up with the best set of result to compare that is available in a database.In the above pipline,the origin is a MySQ

Problem StatementYou’ve been asking everybody you meet the age old question: What is your favorite dinosaur? You’ve been saving their responses in a series of delimited files that have this format.  Tragically, the data you have meticulously gathered is dirty. Before you can execute your master plan to do whatever is that you were planning to do with this information, you must clean your dataset and resave it to a single clean file. The PipelineYou read in the Dino data with the Directory origin.  You remove duplicates on the Dinosaur column using the Record Deduplicator You filter Dinos that have murdered people in the Jurassic Park film series for reasons that are you own using the Stream Selector. Finally, you save your Dino data to one single file on your local system. Your data is clean, filtered and ready for analysis.  If you’re reading this, please comment with your favorite dinosaur. It’s uh...for science. 

This pipeline is populating and updating a Slowly Changing Dimension for the NYC Citibike stations using Transformer for Snowflake. The pipeline is using StreamSets’ Slowly Changing Dimension stage that handles all of the logic for Type 1 and Type 2 dimensions. The SCD stage then produces the correct output along with the metadata that the Snowflake Destination uses to perform the merge.

Transformer for Snowflake: Pipeline to Create Fact TableThis pipeline uses the NYC Citibike data to populate a fact table enriched with station and weather data.  Since this is using Transformer for Snowflake, none of the data leaves the Snowflake Data Cloud since the raw trips data is already loaded in Snowflake! The pipeline generates the SQL that is then executed on Snowflake, inserting the query results into the target tables. The pipeline is using weather data from Weather Source’s free Global Weather & Climate Data for BI data set on the Snowflake Data Marketplace so you can see the weather conditions for each ride.

 Read 911 Event Data and Load to SnowflakeThis pipeline uses the HTTP client to read real time 911 event data from the Seattle Fire Department. The pipeline then flattens a nested JSON object It also uses the latitude and longitude of the incident location to lookup the Census block information from the Census Bureau’s geocoding API.Fun fact: While building this pipeline, an event came through for a dumpster fire currently happening on my block !

This pipeline performs a Reverse ETL process. We take the enriched, coalesced data and then generate the proper JSON payload that is sent to MixPanel to either create or update users.  

Scenario : Processing data in JSON format using Encrypt/decrypt processor and loading it to a destination. Steps to implement the pipeline without receiving Invalid ciphertext type error.1.First Pipeline reads data in JSON format sends it to the Encrypt/decrypt processor with action as Encrypt and load it to Kafka Producer with data format as JSON.Few columns are encrypted using Amazon KMS and fed to Kafka producer.  2.Second pipeline reads data from Kafka Consumer with data format as JSON.Field Type convertor is used to converts columns to Byte Array before feeding data into Encrypt/Decrypt processor because the Encrypt/Decrypt processor will only accept columns with data type as Byte Array.Columns that are converted to Byte Array are sent one by one into a Base64 Field Decoder. Please note that We can pass one field per Base64 Field Decoder.From the Base64 Field Decoders the columns are sent to Encrypt/decrypt processor with action as decrypt and finally the data is written to a file