Welcome

Use Case:  Receive Device data from various machines around the world either using (python scripts/Go lang/MQTT/Kafka/Files). All the data is landed on MQTT Topics. Streamsets pipeline will read the data from the MQTT Topics.Once device data is received from the MQTT Topics, will get more machine details (like Country/City/longitude/Latitude) using Geo IP.Raw data will be stored in S3.Processed Data will be stored into Databricks delta lake and elasticsearch for analysis. Sample analysis:Count all devices for a particular country and map them.Sample Data:battery_level,c02_level,cca2,cca3,cn,device_id,device_name,humidity,ip,latitude,lcd,longitude,scale,temp,timestamp1,234,US,USA,United States,1,meter-gauge-abcdef,10,111.161.225.1,38,green,-97,Celsius,14,1458444054093  Complete Pipeline:  

My name is Brenna and I’m the new Developer Evangelist here at StreamSets. For the majority of my career, I’ve been a data engineer tasked with moving data from various places within an organization into a central data warehouse. I’m sure this is something you all can relate to. I’m really excited to enable you to get the most out of your journey with StreamSets. I’ll be here in the community answering questions, as well as participating in webinars and podcasts and supporting this developer community any way that I can. As I am ramping up here at StreamSets, please share any content or best practices you’ve found. I would love to learn from you too.

    This is a generic pipeline that has been created to  read multiple .csv files from AWS s3 bucket and handles corrupt data if any in the file and masks the data based on DB env and performs full and data load  to the destination DB’s.Let's deep drive into each and every processor which is being used in the pipeline.Origin : AWS S3 : Read multiple .csv files from the bucket.Processors:Expression Evaluator : This is used for setting the pipeline execution time and user to identify who executes the pipeline and when.Field Type Converter:  This is for converting time field  to correct date time format (YYYY-MM-DD HH:MM:SS)Jython Evaluator : This is for handling corrupt data if any in the file (e.g “‘ street address ’”)Stream-sets Selector : This is for checking  for which environment  data needs to be masked before loading data to destination DB’s. In this case for Dev and Test env data will be masked because of GDPR rule we couldn’t load sensitive data into it but for Prod environment

Here's a Kafka Stream Processor pipeline that reads events from a "raw" topic, performs streaming transforms, and publishes the transformed events to a "refined" topic that multiple downstream clients subscribe to: Kafka Stream Processor Pipeline Here is the Stream Processor Pipeline’s placement within a StreamSets Topology that allows visualization and monitoring of the end-to-end data flow, with last-mile pipelines moving the refined events into Delta Lake, Snowflake, Elasticsearch, S3 and ADLS:Kafka Stream Processor with Multiple Consumers Users can extend the Topology using Transformer for Snowflake to perform push-down ETL on Snowflake using Snowpark, and Transformer on Databricks Spark to perform ETL on Delta Lake Push-down ETL on Snowflake and Databricks Delta Lake    

 Hey, everyone. We are very excited to share new automation support for self-managed engine installation using Ansible with our DataOps platform!Why does this project exist?We know there are some folks who love Ansible or other configuration management tools to  automate installation of Streamsets Engines to allow seamless integration with our products. The DataOps Platform helps you to manage all your engines using Environments and Deployments and we want to make it easier for folks to leverage these concepts and automate their workloads.What are the Benefits?DataOps platform’s easy-to-use REST API has been updated and now makes it effortless to integrate with your favorite automation tools. This feature is live on cloud.login.streamsets.com. We've made it possible to create Deployments and start Data Collector and Transformer engines from those Deployments all using an Ansible playbook.How can I get it?To give it a try, clone our sample Ansible playbook on Github, then follow the REA

In this pipeline we solve some fun data problem with the help of  StreamSets Transformer . Transformer execution engine runs data pipelines on Apache Spark. We can run this pipeline on any spark cluster type Problem Statement:  Find the average number of friends for each age and sort them in ascending order.We are given fake friend dataset of social networking platforms in CSV file format is stored in Google cloud storage id, Name, Age, Number of Friends0,Will,33,3851,Jean-Luc,26,22,Hugh,55,2213,Deanna,40,4654,Quark,68,215,Weyoun,59,3186,Gowron,37,2207,Will,54,307   

 MySQL binary logs provide a very efficient way to track data changes. It contain events that describe the modifications to data. In short, binary logs contain all the information that is required to trace the global status of the server right from the time it was started. This pipeline is design to propagates INSERT, UPDATE, and DELETE operations from MySQL BinLog to Snowflake.

Ordnance Survey (OS) produce a suite of products for the UK Market, One of these is Address Base Premium (ABP), Which is a set of details that describe buildings, businesses, residential and items you’d find on a map in detail, such as Lat/Long data, how the royal mail refers to it, and how the local authorities/councils describe and classify those items.Sounds good right? Well its not so nice of a set of data to deal with, It is shipped in 5 km batches of data in a single csv file, across 10 different schema patterns within it.. Nightmare? Nope! You know the secret of streamsets!Streamsets doesn't care about schema on read, That is the key to unlocking this… In the ABP Files, the first column has a record identifier, This tells us which schema and rules to process.This pipeline makes such a difficult issue to deal with normally, clear and transparent.You can watch a record come in from the raw file, read it with no header, Look at the first column. Process that to a given lane, That l

Data comes from a monitor device, with test results of different elements. Those elements have 3 values, their ID, Their Value and any error messages.The customer wants to see a flattened list of just Monitor Device and timestamp with a result of each element on seperate lines.Using the above we can achieve that, we import the data ignoring the header line. (hence labeling them with numbers) First we label the monitor “Parent” fields by using a field renamer We build a Empty map for us to correctly parse the records using expression evaluator  we use a field mapping processor, to map those groups of 3 fields ( checking we only remap columns that are numeric) now we have groups, we split the groups into records using a field pivot processor This leaves a single group per record but as a group, Lets tidy that up with a field flattener so all the records are at the same level  Finally we use a field renamer to label the 3 fields we have produced We ship that off to our secure storage faci

StreamSetters we need your feedback! Do you have 2 mins? We are looking for Community member feedback to help us understand how we're doing and how we can better help you on your journey.   

 Hi Everyone,  Thank you to those who completed the survey. This has been really helpful to understand what we are doing an awesome job on and where we can improve. We wanted to share the responses and pulse of the community today, which you’ll find in the representative responses below.   We will be conducting these surveys on a quarterly basis throughout the year. You can find them on the right side of each community page labeled as “Feedback”.  (REPORT)  Short Answer Question #1:  Thank you for your honest feedback. How can we make the community more helpful for you? Responses  With more documentation of how we implement for streaming and transformer Here are some helpful docs and academy course.  Not able to open ask.streamsets.com As ofDec 8, 2021ask.streamsets.com has been sunset. Going forward Community.streamsets.com is the go-to community forum and knowledge base managed and hosted by StreamSets. We have migrated the top ask.streamsets.com to our new platform and ungat

Please meet our first (Q4 2021) StreamSets Quarterly Community Champion, @Dash! Dash has played a foundational role in building this community, cementing himself a figurehead for data engineering expertise. A previous StreamSets teammate, community builder, technical resource with a wealth of knowledge, fabulous dancer, and explorer–it is a delight to continue to have him within our community. It only makes sense to have our first champion be our StreamSets community leader, advocate, and evangelist. Dash has contributed vital efforts building, assisting, and connecting members within our community. His outreach involves helping members with technical questions, co-hosting the Sources and Destinations podcast, participating in DataOps Summit events, and much more. We can't thank him enough. As of today, Dash sits at the "Headliner" Rank with the creation of 27 Topics,  66 Replies, and solving 40 community member's questions!  Over the last 19+ years, Dash has been hands-on within the s

 This pipeline is designed to capture inserts and updates (SCD Type II) being uploaded to a bucket on Amazon S3 for a slowly changing dimension table -- Customers. The pipeline creates new records with version set to 1 for new customers and with version set to (current version + 1) for existing customers. The customer records are then stored in Snowflake Data Cloud.

 This pipeline is designed to ingest streaming data from Kafka and load a trained ML model in Scala custom processor to predict sentiment of tweets. The pipeline runs on Databricks cluster and stores the tweets along with its score in Delta Lake.

 This pipeline is designed to ingest data from Amazon S3 and prepare it for training a ML model using PySpark custom processor. Once the Gradient Boosted model is trained, the model artifacts, features, accuracy of the model and other metrics are registered as an experiment in MLflow. (The pipeline runs on Databricks cluster which comes bundled with MLflow server.) 

 This pipeline is designed to handle (embrace!) data drift while ingesting web logs from AWS S3 and then transforming and enriching them before storing the curated data in Snowflake Data Cloud. The data drift alert is triggered if/when the data being ingested is missing a key field IP Address which is crucial for downstream analytics. 

Currently the platform does not allow you the members to change this on your own. The team is looking to allow this functionality. I know this is frustrating. To work around this, please email Drew.Kreiger@streamsets.com to change your email and or other profile questions/needs.  -Drew 

Happy New Years everyone!  I want to say thank you to each of you for all your hard work and contributions to our StreamSets Community. We are close to welcoming our 300th member! Taking a look back at last year we launched our new community platform with the addition of knowledge base articles, a monthly community-led newsletter, bi-annual member feedback survey, a new and improved StreamSets Academy, and much much more. Looking ahead into 2022 we are excited to launch; Virtual meetups, Pipelines and patterns examples, and much more from the feedback we received from the member feedback survey. (Click image to expand) 

Hi everyone,  As of December 8th 2021 we have sunset ask.streamsets.com. You may have some questions and this post is to help answer.  Why did we close down ask.streamsets.com? As we continue to grow our community the ask.streamsets.com website did not meet our level of support we wanted to provide to the community. What content was migrated over? The top 30 relevant topics were migrated over. Most questions you may have that once was found at  ask.streamsets.com can be found in our up to date docs. https://streamsets.com/support/documentation-overview/Also, at this same time we have un-gated and made free over 400 knowledge base articles from our support portal.  Do I maintain my status and points once had on ask.streamsets.com?Sorry, no. We wish the systems allowed this function. Going forward, Please check out our leaderboard https://community.streamsets.com/leaderboard?period=thisWeek and  Where do I post my questions going forward? Post your questions and start a conversations her

We are excited to share that we have added a new theme to our badges and ranks within the StreamSets Community Platform. Let's go over what's new while also explaining how to earn points, badges, and ranks to reach Rock Star status within the StreamSets Community.   StreamSets Community ThemeWe believe we Rocked the theme. 😏 We have designed our community around experts like yourself  ranking up to become Data Rock Stars. Today, we have added a new category banner image and rank icons following members reaching the Rock Star status. We hope you like the new look.   What are badges? Badges are a great way to share and highlight a user of a specific milestone or involvement within the community. Badges range from; completing a course on StreamSets Academy, becoming StreamSets certified, speaking at DataOps Summit, and becoming a guest on our Sources and Destinations data engineering podcast.  Take a Peep at the nifty badges! (only a few here)To earn a badge, sign up for StreamSets Acade

 Hi everyone, This post will co-inside with the September 28th Demos with Dash event! Please ask a question, share what you enjoyed or disliked.  Dash is excited to chat further!

Hello everyone, Welcome to the StreamSets Community. I am excited to help empower members as they continue their journey to learn, share, and grow their knowledge to succeed each day. I want to introduce myself, and I hope to read more about you too!I am Drew Kreiger. I recently started as the Senior Community Manager here at StreamSets in April of 2021. In the past 5 years, I have previously worked with communities at Talend and now called Redis, where I managed community meetups, education programs, hackathons, forums, and many other great programs with fantastic community members.I enjoy working with community members as I enjoy helping users overcome issues/challenges. I also enjoy working with users on community content and seeing the impact a blog, podcast, KB article, and or event has made within the community. A fun fact about me. During college, I had studied to become a sommelier. Cheers!

Hi everyone, You might be asking, what is the difference with a question and conversation? That is a great question.A conversation topic is used when you want to share something and involve the community into a discussion.A question topic is used when you need a solution for your question or problem from your community peers. I hope this helps. Lets go create questions and conversations!

We want to make sure your questions are seen and answered. If you ask your question the right way, we will accomplish answering your question fast and precisely. Here are some tips: 1. Before you ask, search first! Make sure to search your question first. The search icon/bar will always be found:  At the top of the home page.  Within creating a new topic.  Within a topic post near your profile photo (Right Top Corner) 2. Don't hesitate ASK! This is a judge-free community. We are all here to support others and build our StreamSets knowledge.  3. Keep your data yours 🧑‍We cannot stress this enough. Do not share any persons' or your personal information (Email, Phone Number, Address, banking info, etc.) in a screenshot, post, messages, or anywhere on the StreamSets Community Platform. 4. Provide all information Please be concise with your topic/ question Title. Regarding your topic/ question description, Please be sure to provide as much detail; Platform, Version, Screenshots, Categories

Join and listen to our latest episode of our Sources and Destinations podcast from our hosts, @iamontheinet and Sean Anderson.S&D is a podcast about data engineering and data science talking about common design patterns and best practices. Listen where ever you get your podcasts. https://linktr.ee/sourcesanddestinations