Welcome

ENV:-SDC is integrated with CDH/CDP. Kerberos is enabled. start event script/end event scripts are being used to get the kerberos credentials.ISSUE:-start event script/end event scripts try to kinit and end up with following error message:-ERROR stderr: kinit: Pre-authentication failed: Permission denied while getting initial credentials Resolution:-By default CDH/CDP enabled clusters has set environment variable KRB5CCNAME add following line to at the beginning of the script to unset the variable temporarilyunset KRB5CCNAME

ERROR:- failed to store credentials: credentials cache permissions incorrect(krb5cc_cldr)

10 months ago

john.mcavoyStreamSets Employee

Problem DescriptionWhen running an SDC pipeline which processes records with Strings containing UTF-8 special characters, these special characters are being replaced by question marks (? or �) in various parts of your pipeline.ExampleInput record:{"productName": "My Product™"}Output record:{"productName": "My Product�"}Root CauseThis problem indicates that your Java Runtime Environment is using a non-UTF-8 character set which does not support these special characters, so the special characters are being replaced by the � (U+FFFD) REPLACEMENT CHARACTER.By default, SDC will try to set your JAVA_OPTS to use the UTF-8 encoding, however it is possible that the JAVA_OPTS parameters are being set outside of SDC and are overriding either of the following JVM parameters which tells the JRE what encoding to use at runtime: file.encoding and sun.jnu.encoding.The JRE also has a default character set which it will use if these two parameters are not specified. This default character set can vary be

UTF-8 Special Characters being replaced by Question Marks (�)in Pipelines

5 months ago

rvlozanoStreamSets Employee

Question:How to connect JDBC Consumer Stage to MS SQL Server via AAD or Active Directory? Answer:Our product teams are currently evaluating to see this could be added natively. Also, our Product Teams has this on the future Product Road Map for native support.For now you will get the following error message if you try to use “authentication=ActiveDirectoryPassword” in your JDBC URL.JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Failed to load MSAL4J Java library for performing ActiveDirectoryPassword authentication.: hikariConfigBean.connectionStringThis is because the out of box Microsoft SQL Server Drivers comes directly from Microsoft. Microsoft does not package support directly for the authentication method ActiveDirectoryPassword. For this to work you must acquire MSAL4J and it’s dependencies and add them to SDC’s java classpath for the given JDBC Stage. You can add the jars via the pipelin

How to connect JDBC Consumer Stage to MS SQL Server via AAD?

11 months ago

Drew KreigerSenior Community Builder at StreamSets

New to the StreamSets Community? If so, this is the place. Welcome! First thing first: We are here for YOU. If you are a first-time StreamSets user, customer, partner or a StreamSets Rock star, we have you covered. The StreamSets community is the go-to place to learn, share, and grow your knowledge with other like-minded professionals. If you want to go quickly, go alone. If you want to go far, go together. - African proverb Take a tour of the general welcome community posts.-Please head over to our community guidelines page to learn:Who we are a company and a community team Unacceptable Behavior/ Guidelines (Do’s and Dont’s) Code Of Conduct Communication Guidelines Reporting Harassment or Other IssuesOther Platform related helpful topics: 5 tips to ask your question the right way Question vs Conversation Meet the Community | Introduce yourself Community Platform Theme, Bages, and Ranks DocumentationLearn how to make the most of StreamSets with user guides and tutorials here StreamS

New to the StreamSets Community? Everything to get you started. 🚀

Drew KreigerSenior Community Builder at StreamSets

Streamsetters! 2 Community members will each earn a $250 Amazon Gift card by submitting the most impressive or creative Pipelines. Please share your pipeline within this category (Show us your Pipeline). It's easy to submit, Make sure to: Create a new Topic in the Show us your Pipeline category. (Each submission needs to have its own topic thread) Add an image of your sample pipeline. (MAKE SURE NO PERSONAL,CREDENTIALS, OR COMPANY INFO IS SCREEN SHOT OR SHARED) Add an explanation of 2-3 sentences of your pipeline use case and benefits. Lastly, Make it fun! Create something useful, fun, creative, and or wacky. The closing date of our 1st “Share your Pipelines” is March 31, 2022 Example Entry:By @Mike Carley Here are other sample Pipelines. https://github.com/streamsets/pipeline-libraryHow we will choose the three winners: Winner will be announced on Mar 31, 2022. The two winners will be graded on overall impressions, creativity, originality, and explantation. Winners will be

Share your Pipeline for a chance to win a $250 Amazon Gift Card! Example here!

11 months ago

swayamDiscovered Fame

Dear Community users,To extend the last topic where I shared how to convert the CSV to JSON, what if you have to combine multiple JSON records in a batch into a single JSON?I often came with requirements where we have to call an external API which suports both single as well as batch requests and most often batch is preffered to avoid the multiple round-trips and increae performance. This topic can be very useful for developers who wants to convert the multiple CSV records coming in a batch to a single JSON list array and send them as a single record to a target destination.Problem StatementConvert the 2 input CSV records into a single JSON.Input CSV Records:NAME|AGE|DEPARTMENT|ADDRESS"John"|"23"|"HR"|"31, 2nd Cross, 4th Street, Wilson-543234""Amanda"|"26"|"HR"|"24, 1st Cross, Wilson Garden-543234"Output Single JSON Record:[ { "data": { "NAME": "John", "AGE": "23", "DEPARTMENT": "HR", "ADDRESS": "31, 2nd Cross, 4th Street, Wilson-543234" } }, { "data": {

Swayam's Micro-blogging: Combine multiple JSON records into a single one

swayamDiscovered Fame

Hi,The above pipeline reads the CSV files uploaded into an S3 bucket for new contact file uploads. Based on the new contacts available as a CSV, it converts them into a JSON. But the critical part here is that it merges each batches (each batch size is 1000) into a single JSON of multiple records and finally make a single HTTP POST call to one of our internal system to carry out a CONTACT update call.

Convert multiple CSV records into a single JSON and make an HTTP bulk API POST call

6 months ago

swayamDiscovered Fame

Dear Community users,SDC offers a great out-of-box processor “Field Type Converter” to convert an incoming field of one data type to another data type. This is a very useful transformation function that is required before we send the data to final destination. However, sometime we may have to write our own transformation function and today I’m going to share a scenario where I was forced to use a jython evaluator to detect the date format of an incoming field and convert that to a single unified format that our destination is expecting.Problem StatementWe have a need to process the incoming file where there is a field that need to be processed as “date” data type while there is another field which needs to be processed as “timestamp” data type before we generate the final feed for destination. The detaination while accepts the “date” fields is YYYY-MM-DD format, the timestamp fileds need to be of YYYY-MM-DD HH:MI:SS GMT+05:30, the origin fields can be of different formats. The solution

Swayam's Micro-blogging: Auto detection of date format

AkshayJadhavStreamSets Employee

Environment:Streamsets Data Collector 5.1. Producer or consumer Library for Microsoft SQL Server. Issue: The pipeline starts failing with following messages when try to fetch or produce the data to Microsoft SQL Server.This happens on data collector is 5.1 but the same pipeline is working fine on 4.x versions.com.microsoft.sqlserver.jdbc.SQLServerException: The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: "sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target". ClientConnectionId:af15a6cf-f28d-4e21-adb3-141826becad0 at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:3680) ~[mssql-jdbc-10.2.1.jre8.jar:?] at com.microsoft.sqlserver.jdbc.TDSChannel.enableSSL(IOBuffer.java:2113) ~[mssql-jdbc-10.2.1.jre8.jar:?] at com.microsoft.sqlserver.jdbc.SQLServer

Connecting to Microsoft SQL Server from SDC v5.1 is failing "unable to find valid certification path to requested target"

6 months ago

N_aliOpening Band

I have some json objects that looks as follows and i want them to be all merged in an array like :data: [{nstaskid: “...”,awb: “….”},{nstaskid: “...”,awb: “….”},] i tried to use javascript evalutor , loop over the records and push them in an array and write this record array , but it didn’t work.Can you please help me with that .

aggregate the records into a single array of objects

6 months ago

BikramHeadliner

This is a generic pipeline that has been created to read multiple .csv files from AWS s3 bucket and handles corrupt data if any in the file and masks the data based on DB env and performs full and data load to the destination DB’s.Let's deep drive into each and every processor which is being used in the pipeline.Origin : AWS S3 : Read multiple .csv files from the bucket.Processors:Expression Evaluator : This is used for setting the pipeline execution time and user to identify who executes the pipeline and when.Field Type Converter: This is for converting time field to correct date time format (YYYY-MM-DD HH:MM:SS)Jython Evaluator : This is for handling corrupt data if any in the file (e.g “‘ street address ’”)Stream-sets Selector : This is for checking for which environment data needs to be masked before loading data to destination DB’s. In this case for Dev and Test env data will be masked because of GDPR rule we couldn’t load sensitive data into it but for Prod environment

Read multiple csv from S3,Handle corrupt data,Mask the sensitive data and send Data to on premise and Denodo DB

Drew KreigerSenior Community Builder at StreamSets

Welcome to the StreamSets Community! Please read over the following Community Guidelines and Code of Conduct below. Please also view StreamSets work culture Blog and Privacy policy. Who we areThe StreamSets Community Is a global community of data engineers who set out to learn, share, and expand their DataOps skills. “If you want to go fast, go alone. If you want to go far, go together” – African Proverb GuidelinesThe Do’s and Don’ts Do’s: Share helpful docs/resources, tips & tricks, and useful questions Network with other StreamSets users Help answer questions via the forum Challenge and push boundaries Be positive, Be kindDon’ts: Share personal information Attack/Call out others Degrade other members based on their knowledge level Promote a personal brand or company Code Of Conduct The StreamSets Community is dedicated to providing a harassment-free, equitable and inclusive community experience for everyone. We do not tolerate harassment of members in any form. We take this po

StreamSets Community Guidelines

sujatha mogiliFan

The streamsets pipeline fails on a regular basis with the below error. Please advise if you ran into this issue and the resolution. We have increased Heap size couple of times but not helping.“ERROR A JVM error occurred while running the pipeline, java.lang.OutOfMemoryError: Java heap space”

The streamsets pipeline fails with java.lang.OutOfMemoryError: Java heap space

7 months ago

swayamDiscovered Fame

PrefaceFetching online and offline data across different channels to understand your end-users behavior is a primary topic for any enterprise-level brand. Customer 360 view of an end customer gives enormous insight to marketing leaders who can then prepare their strategy to engage and entice these users. A full-stack CDP (Customer Data Platform) helps meet the needs of marketing leaders and analytic folks.A stable, reliable, and scalable data processing stack is the de facto need for any enterprise CDP solution, and to set up that, data ingestion and extraction architecture need to be solid. StreamSets Data Collector (SDC) has become the preferred tool for many data engineers and data stewards to achieve the ETL and ELT-related needs in this area.In the traditional approach, the ETL engineers normally have definitive requirements in a place where they know the origin, transformation rules, and destination. Based on the requirements, engineers develop their pipelines and deploy them to

Building a No-code Data Connector Platform Using StreamSets Data Collector REST APIs

swayamDiscovered Fame

Problem StatementVery often we find duplicate records being sent by customer while exporting data from their DWH and sending to us. To ensure data quality, we need to filter out the duplicate records before they enter into our platform for processing, computation and analysis.SolutionWe have addressed this using 2 separate pipelines.Pipeline-1: Files arriving in S3 gets picked up by this pipeline where for every record we calculate the MD5 Hash and make an entry in REDIS with KEY as “The MD5 hash value of the record” and VALUE as “Filename-RowNumber” of the current record being processed. Pipeline-2: The same file processed by Pipeline-1, again gets picked by Pipeline-2 and for every record it calculates the MD5 Hash and checks it’s value in REDIS to determine which RowNumber we need to process and which one we should drop as it’s a duplicate record.Pipeline-1: Pipeline-1: Calculating Hash and Inserting to RedisAs you may see above:CSV file picked from S3 origin, gets converted to JSO

Handling Duplicate Data

wilsonshamimStreamSets Employee

Use Case: Receive Device data from various machines around the world either using (python scripts/Go lang/MQTT/Kafka/Files). All the data is landed on MQTT Topics. Streamsets pipeline will read the data from the MQTT Topics.Once device data is received from the MQTT Topics, will get more machine details (like Country/City/longitude/Latitude) using Geo IP.Raw data will be stored in S3.Processed Data will be stored into Databricks delta lake and elasticsearch for analysis. Sample analysis:Count all devices for a particular country and map them.Sample Data:battery_level,c02_level,cca2,cca3,cn,device_id,device_name,humidity,ip,latitude,lcd,longitude,scale,temp,timestamp1,234,US,USA,United States,1,meter-gauge-abcdef,10,111.161.225.1,38,green,-97,Celsius,14,1458444054093 Complete Pipeline:

IOT Usecase : Collect Device data to Databricks Delta lake/Elasticsearch

amarzeetFan

I want to use stream set data collector legacy version (3.x) on a linux machine. Only option is to install the tar ball. I couldn't find a link on streamset website to download and use it. Can you please provide this information

Streamset data collector 3.x tarball download link not available

2 months ago

Drew KreigerSenior Community Builder at StreamSets

published in Community Newsletter

Community Newsletter 1st Edition: Members Recognition is here!

Welcome to our very first Community Newsletter. This year our community has been hard at work so we are excited to announce the 2021 DataOps Summit Community Awards. Please join me in congratulating Joseph Arriola, Saleem Pothiwala, and Naveen Siddareddy for all their great work towards the community including contributions -- Sources & Destinations Podcast, DataOps Summit, blogs, and much more! If you would like to watch the awards ceremony or any of our other Summit sessions, register and watch the on-demand content here. https://www.dataopssummit-sf.com/aboutThis newsletter will keep you up today on the latest from the StreamSets Community--exciting topics, community members’ milestones, highlights, and also broader industry news. If you would like to share a helpful blog, industry article, or other information you think the community could benefit from, email us at community@streamsets.comMore exciting news to share! We will be highlighting one StreamSets Community member as th

1300

Drew KreigerSenior Community Builder at StreamSets

Thank you to the 24 community members who submitted a pipeline! Our two winners for this quarters Share your Pipeline Contest is @swayam and @Bikram! (Stay on the look out for a DM from me.) Check out Swarm’s example pipeline, Carrying out image comparision using SDC pipeline and Amazon Image Rekognition. Where he compares the images received from facebook and twitter to determine if the 2 different profiles belongs to same customer or not. Bikram example Pipeline is, Read multiple csv from S3,Handle corrupt data,Mask the sensitive data and send Data to on premise and Denodo DB. His pipeline reads multiple .csv files from AWS s3 bucket and handles corrupt data if any in the file. The pipeline handles this while masking the data based on DB env and performs full data loads to the destination DB’s. Stay on the look out for our next Share your Pipeline Contest! We are looking to use Miroverse for ease of trying new example pipeline via one click.

Announcement: Share your Pipeline Contest Winners! 🏆

11 months ago

Drew KreigerSenior Community Builder at StreamSets

posted in Community Newsletter

Community Newsletter 2nd Edition: Thanks for all your Contributions

Community Newsletter November edition is here! As we take a quick glance back throughout 2021, we as a community have completed, gained, and accomplished a lot! From the entire team at StreamSets, thank you for your community contributions. In next month's newsletter we will announce and showcase our Community Champion of the Quarter. Each quarter, our Community Champion will earn:One free StreamSets Academy certification Exam Their very own Champion badge A personal highlight piece, and Custom SWAG!There’s still time to contribute and land your very own Community Champion title.What's NewStreamSets Academy is now on the Community Platform 5-10 new Knowledge Base articles will be added weekly in the coming weeks Contribute and publish your articles HERE to become a KB article by showcasing your usage, architecture, and know-how of the StreamSets DataOps Platform. Watch out! They just might end up seen as a Knowledge Base article as well. Snowflake and StreamSets’ Partnership Means Acce

Drew KreigerSenior Community Builder at StreamSets

StreamSetters we need your feedback! Do you have 2 mins? We are looking for Community member feedback to help us understand how we're doing and how we can better help you on your journey.

Do you have 2 Minutes? We need your feedback!

subashiniStreamSets Employee

Scenario : Processing data in JSON format using Encrypt/decrypt processor and loading it to a destination. Steps to implement the pipeline without receiving Invalid ciphertext type error.1.First Pipeline reads data in JSON format sends it to the Encrypt/decrypt processor with action as Encrypt and load it to Kafka Producer with data format as JSON.Few columns are encrypted using Amazon KMS and fed to Kafka producer. 2.Second pipeline reads data from Kafka Consumer with data format as JSON.Field Type convertor is used to converts columns to Byte Array before feeding data into Encrypt/Decrypt processor because the Encrypt/Decrypt processor will only accept columns with data type as Byte Array.Columns that are converted to Byte Array are sent one by one into a Base64 Field Decoder. Please note that We can pass one field per Base64 Field Decoder.From the Base64 Field Decoders the columns are sent to Encrypt/decrypt processor with action as decrypt and finally the data is written to a file

Solving Invalid Cipher text issue with Kafka Stages using JSON Data format

Drew KreigerSenior Community Builder at StreamSets

We are excited to share that we have added a new theme to our badges and ranks within the StreamSets Community Platform. Let's go over what's new while also explaining how to earn points, badges, and ranks to reach Rock Star status within the StreamSets Community. StreamSets Community ThemeWe believe we Rocked the theme. 😏 We have designed our community around experts like yourself ranking up to become Data Rock Stars. Today, we have added a new category banner image and rank icons following members reaching the Rock Star status. We hope you like the new look. What are badges? Badges are a great way to share and highlight a user of a specific milestone or involvement within the community. Badges range from; completing a course on StreamSets Academy, becoming StreamSets certified, speaking at DataOps Summit, and becoming a guest on our Sources and Destinations data engineering podcast. Take a Peep at the nifty badges! (only a few here)To earn a badge, sign up for StreamSets Acade

Community Platform Theme, Badges, and Ranks

javierdallamoreFan

Overview Since November 2014 we have been implementing a Data Lake platform to collect, process, filter, enrich, store, analyse and visualize logs and business events using Big Data and streaming technologies. The main goal is to detect important events, generate statistics and produce alerts to solve problems in a proactive way.Our first pilot on a new customer is usually related to application logs but once this platform is up and running it naturally becomes the centralized repository not only for logs but also for any type of sources and data formats. With dashboards on display in big screens other areas and departments start requesting dashboards of their own that require new data sources to be connected.To adapt to different use cases and new requirements we needed a more modular, flexible and extensible architecture. As the Data Lake gained prominence within our firsts customers and other project opportunities materialized, challenges and constraints emerged that led us to rethi

Our Journey Finding the Perfect-Fit Data Integration Tool

SusannahStreamSets Employee

Hey, everyone. We are very excited to share new automation support for self-managed engine installation using Ansible with our DataOps platform!Why does this project exist?We know there are some folks who love Ansible or other configuration management tools to automate installation of Streamsets Engines to allow seamless integration with our products. The DataOps Platform helps you to manage all your engines using Environments and Deployments and we want to make it easier for folks to leverage these concepts and automate their workloads.What are the Benefits?DataOps platform’s easy-to-use REST API has been updated and now makes it effortless to integrate with your favorite automation tools. This feature is live on cloud.login.streamsets.com. We've made it possible to create Deployments and start Data Collector and Transformer engines from those Deployments all using an Ansible playbook.How can I get it?To give it a try, clone our sample Ansible playbook on Github, then follow the REA

Automation using Ansible with the latest version of our Control Hub (Control plane)