Show us your Pipelines

Share an example of your Pipeline pattern. Upload a Screenshot and detailed description. No wrong answers. Do NOT share personal or credential information.

  • 62 Topics
  • 71 Replies

62 Topics

onefoursix
StreamSets Employee
onefoursixStreamSets Employee
posted in  Show us your Pipelines

Kafka Stream Processor combined with Push-down ETL on Snowflake and Databricks Delta Lake

Here's a Kafka Stream Processor pipeline that reads events from a "raw" topic, performs streaming transforms, and publishes the transformed events to a "refined" topic that multiple downstream clients subscribe to: Kafka Stream Processor Pipeline Here is the Stream Processor Pipeline’s placement within a StreamSets Topology that allows visualization and monitoring of the end-to-end data flow, with last-mile pipelines moving the refined events into Delta Lake, Snowflake, Elasticsearch, S3 and ADLS:Kafka Stream Processor with Multiple Consumers Users can extend the Topology using Transformer for Snowflake to perform push-down ETL on Snowflake using Snowpark, and Transformer on Databricks Spark to perform ETL on Delta Lake Push-down ETL on Snowflake and Databricks Delta Lake    

0
onefoursix
StreamSets Employee
2 years ago
dima
StreamSets Employee
dimaStreamSets Employee
posted in  Show us your Pipelines

Analyzing chess games with batch and streaming pipelines

I wanted to have a way to send chess game information from lichess.org (a popular chess server) to Elasticsearch and Snowflake to let me visualize statistics (e.g., how often does World Champion GM Magnus Carlsen lose to GM Daniel Naroditsky?) as well as to generate reports in Snowflake (e.g., when Magnus does lose, which openings does he tend to play?). I ended up accomplishing this with two pipelines: This pipeline is a batch pipeline that ingests data using the lichess REST API and pushes it to a Kafka topic. It uses an endpoint that allows for pulling down all games by username, so I’ve parameterized the username portion and then can use Control Hub jobs to pick which specific users I want to study. This pipeline consumes game data from my Kafka topic, does some basic cleanup of the data (adds a field for the name of the winner rather than the color of the winning pieces and converts timestamps from long to datetime) as well as some basic enrichment (it adds a field that calculates

1
R
Fan
3 months ago
Rishi
StreamSets Employee
RishiStreamSets Employee
posted in  Show us your Pipelines

Solving Fun Data problem with Streamsets transformer

In this pipeline we solve some fun data problem with the help of  StreamSets Transformer . Transformer execution engine runs data pipelines on Apache Spark. We can run this pipeline on any spark cluster type Problem Statement:  Find the average number of friends for each age and sort them in ascending order.We are given fake friend dataset of social networking platforms in CSV file format is stored in Google cloud storage id, Name, Age, Number of Friends0,Will,33,3851,Jean-Luc,26,22,Hugh,55,2213,Deanna,40,4654,Quark,68,215,Weyoun,59,3186,Gowron,37,2207,Will,54,307   

0
Rishi
StreamSets Employee
2 years ago
antmcmullen
StreamSets Employee
antmcmullenStreamSets Employee
posted in  Show us your Pipelines

Ordinance Survey - Address Base Premium Processing mutli-schema documents easily

Ordnance Survey (OS) produce a suite of products for the UK Market, One of these is Address Base Premium (ABP), Which is a set of details that describe buildings, businesses, residential and items you’d find on a map in detail, such as Lat/Long data, how the royal mail refers to it, and how the local authorities/councils describe and classify those items.Sounds good right? Well its not so nice of a set of data to deal with, It is shipped in 5 km batches of data in a single csv file, across 10 different schema patterns within it.. Nightmare? Nope! You know the secret of streamsets!Streamsets doesn't care about schema on read, That is the key to unlocking this… In the ABP Files, the first column has a record identifier, This tells us which schema and rules to process.This pipeline makes such a difficult issue to deal with normally, clear and transparent.You can watch a record come in from the raw file, read it with no header, Look at the first column. Process that to a given lane, That l

0
N
5 months ago
antmcmullen
StreamSets Employee
antmcmullenStreamSets Employee
posted in  Show us your Pipelines

Split child records that are crosstabbed back into flat records

Data comes from a monitor device, with test results of different elements. Those elements have 3 values, their ID, Their Value and any error messages.The customer wants to see a flattened list of just Monitor Device and timestamp with a result of each element on seperate lines.Using the above we can achieve that, we import the data ignoring the header line. (hence labeling them with numbers) First we label the monitor “Parent” fields by using a field renamer We build a Empty map for us to correctly parse the records using expression evaluator  we use a field mapping processor, to map those groups of 3 fields ( checking we only remap columns that are numeric) now we have groups, we split the groups into records using a field pivot processor This leaves a single group per record but as a group, Lets tidy that up with a field flattener so all the records are at the same level  Finally we use a field renamer to label the 3 fields we have produced We ship that off to our secure storage faci

0
antmcmullen
StreamSets Employee
2 years ago
Rishi
StreamSets Employee
RishiStreamSets Employee
posted in  Show us your Pipelines

CDC pipeline from Mysql to Snowflake

 MySQL binary logs provide a very efficient way to track data changes. It contain events that describe the modifications to data. In short, binary logs contain all the information that is required to trace the global status of the server right from the time it was started. This pipeline is design to propagates INSERT, UPDATE, and DELETE operations from MySQL BinLog to Snowflake.

1
Drew Kreiger
Rock star
2 years ago
Dash
Headliner
DashSenior Technical Evangelist and Developer Advocate at Snowflake
posted in  Show us your Pipelines

Slowly but surely!

 This pipeline is designed to capture inserts and updates (SCD Type II) being uploaded to a bucket on Amazon S3 for a slowly changing dimension table -- Customers. The pipeline creates new records with version set to 1 for new customers and with version set to (current version + 1) for existing customers. The customer records are then stored in Snowflake Data Cloud.

0
Dash
Headliner
2 years ago
Dash
Headliner
DashSenior Technical Evangelist and Developer Advocate at Snowflake
posted in  Show us your Pipelines

Real-time scoring using ML model

 This pipeline is designed to ingest streaming data from Kafka and load a trained ML model in Scala custom processor to predict sentiment of tweets. The pipeline runs on Databricks cluster and stores the tweets along with its score in Delta Lake.

0
Dash
Headliner
2 years ago
Dash
Headliner
DashSenior Technical Evangelist and Developer Advocate at Snowflake
posted in  Show us your Pipelines

Train ML Model and register experiment in MLflow

 This pipeline is designed to ingest data from Amazon S3 and prepare it for training a ML model using PySpark custom processor. Once the Gradient Boosted model is trained, the model artifacts, features, accuracy of the model and other metrics are registered as an experiment in MLflow. (The pipeline runs on Databricks cluster which comes bundled with MLflow server.) 

0
Dash
Headliner
2 years ago
Dash
Headliner
DashSenior Technical Evangelist and Developer Advocate at Snowflake
posted in  Show us your Pipelines

Orchestrate Initial Bulk Load to Change Data Capture

 This pipeline is designed to orchestrate initial bulk load to change data capture workload from on-prem Oracle database to Snowflake Data Cloud. The pipeline takes into consideration the completion status of the initial bulk load job before proceeding to the next (CDC) step and also sends success/failure notifications. The pipeline also takes advantage of the platform’s event framework to automatically stop the pipeline when there’s no more data to be ingested. 

1
P
Opening Band
1 year ago
Dash
Headliner
DashSenior Technical Evangelist and Developer Advocate at Snowflake
posted in  Show us your Pipelines

Embrace Data Drift or get left behind!

 This pipeline is designed to handle (embrace!) data drift while ingesting web logs from AWS S3 and then transforming and enriching them before storing the curated data in Snowflake Data Cloud. The data drift alert is triggered if/when the data being ingested is missing a key field IP Address which is crucial for downstream analytics. 

0
Dash
Headliner
2 years ago
Drew Kreiger
Rock star
Drew KreigerSenior Community Builder at StreamSets
posted in  Show us your Pipelines

Share your Pipeline for a chance to win a $250 Amazon Gift Card! Example here!

Streamsetters!  2 Community members will each earn a $250 Amazon Gift card by submitting the most impressive or creative Pipelines. Please share your pipeline within this category (Show us your Pipeline).  It's easy to submit, Make sure to: Create a new Topic in the Show us your Pipeline category. (Each submission needs to have its own topic thread) Add an image of your sample pipeline. (MAKE SURE NO PERSONAL,CREDENTIALS, OR COMPANY INFO IS SCREEN SHOT OR SHARED) Add an explanation of 2-3 sentences of your pipeline use case and benefits. Lastly, Make it fun! Create something useful, fun, creative, and or wacky.   The closing date of our 1st “Share your Pipelines” is March 31, 2022 Example Entry:By @Mike Carley Here are other sample Pipelines. https://github.com/streamsets/pipeline-libraryHow we will choose the three winners: Winner will be announced on Mar 31, 2022.  The two winners will be graded on overall impressions, creativity, originality, and explantation. Winners will be

4
Drew Kreiger
Rock star
2 years ago

Badge winners

Show all badges
Terms & Conditions