- - Knowledge base
Product Updates
Events

30-Day Free Trial: It’s Never Been Easier To Get Started With StreamSets

4 months ago

Home
Community overview
Welcome

Welcome

Welcome to the StreamSets Community

76 Topics

Recently active

Most replies Most views Newest first

NCollinsStreamSets Employee

published in Welcome Guide

NCollinsStreamSets Employee

published in Welcome Guide

Wisdom of Crowds End-User Survey - Participate Now!Blog

Participate in the 2023 Wisdom of Crowds mid-year research process and receive a complimentary report of study findings and a gift from StreamSets as a thank you. Dresner Advisory Services is inviting all business and IT users to participate in its annual examination of the state of the BI, analytics, and information infrastructure marketplaces focusing on drivers, usage, and products. The 2023 mid-year survey covers Analytical Data Infrastructure (ADI), as well as Embedded BI, Data Engineering, Workforce Planning and Analysis, and Supply Chain Planning and Analysis. Participants only need to fill in areas that are relevant to their current usage and experience. Users in all roles and throughout all industries are invited to contribute their insight via an online survey. The corresponding Dresner Advisory Services research will be published beginning in 4Q23 and throughout 1Q24. Qualified survey participants will receive complimentary copies of the findings. Click here to start the

2 months ago

lakshmi_narayanan_tDiscovered Fame

asked in Show us your Pipelines

how to use Google adress api to validate adress column or records

I want to validate the address column by using google address api how to do that .this is my record : I want to validate the address .how to do any one can help this scenario.

2 months ago

NCollinsStreamSets Employee

published in Welcome Guide

Join the StreamSets Product Advisory CouncilBlog

Do you have ideas and want to help shape StreamSets products? Join the Product Advisory Council and innovate WITH us! To sweeten the deal, you can even get rewarded for being a member. 🎉To join and learn more about the council, visit https://streamsets.com/product-advisory-council/

2 months ago

shaik111Fan

asked in Show us your Pipelines

Issue with TCP Server Origin listening port

Am trying to listen some of the port from tcp server origin but its not giving any results, the same port am when using in linux console which we deploy streamsets server am able to listen the message currently am using Version:StreamSets Data Collector 3.19.1 in log am getting TCP_00 - Cannot bind to port [0.0.0.0/0.0.0.0:******]: java.net.BindException: Address already in usehere am hiding port number for confidential with *Could you please help me how to listen messages from streamsets tcp server origin. Thanks in advance

2 months ago

kanyewesthoodieeFan

asked in Show us your Pipelines

How can I compare the results of training?

Hello, CommunityI'm developing a training session. I would need a PRE-training survey and a POST-training survey for this training. The questions (and responses) on both surveys are the same. Ideally, I'd like to show the audience the changes from both the PRE and POST surveys, demonstrating the training's effectiveness on a single screen or report. Additionally, I'd like to provide more fundamental information (such as the average score, the minimum and maximum, the standard deviation, etc.).Does this even exist as a feature, or does anyone know how to accomplish it? I bought the Professional Plan but need help figuring out how to do this. Or does anyone have any other ideas about how to demonstrate improvement?

3 months ago

NCollinsStreamSets Employee

published in Welcome Guide

Write a review & receive a $10 gift cardBlog

Hi StreamSets Community! We want to hear from you and ask if you’d share your experience using StreamSets on Capterra. When you write an honest review of your experience on Capterra, the first 30 qualifying reviewers will receive a $10 gift card as a thank you. Your review will help users of technology products and services consider us as their partner for solving their data integration challenges.We appreciate your honest feedback and thank you for your time.Submit your review now

110

3 months ago

NCollinsStreamSets Employee

published in Events & Webinars

Sneak peek at the next Build It with Brenna - check it out!

The next Build It with Brenna event is happening Wednesday, August 30th. And Brenna is inviting you to join her! Don’t miss it and register now!

110

3 months ago

realmatchaOpening Band

asked in Show us your Pipelines

Connect SqlServer

Hi, I am new to Streamsets, started using it today.I have a task to create some backup data from sqlserver to hadoop and hive, the jdbc connection has been changed to sqlserver but it still doesn't work, please help me

3 months ago

HIMANSHU_SURANAFan

asked in Show us your Pipelines

Consuming MySQL binlog data in JDBC producer

I am trying to build CDC pipeline to migrate MySQL database to MySQL database in different server.Here’s the data collector pipeline I’ve created. As per the documentation here, JDBC producers should be able to process binlog data. I’ve used Field remover to only use /Data and /Table fields. When I run the pipeline I am getting error that input record has no data for <schema>.<table> . How can I create a pipeline to consume binlog records?

3 months ago

SubhaFan

asked in Show us your Pipelines

HTTP Server

Can anyone let me know on how to use the HTTP Server Origin stage to fetch large volume of recordsMy problem statement is, I need to fetch large volume of data from a source using GET call. We can use HTTP Client stage for the same but since the source application doesn't have pagination configured , we will receive all the records at once . Thus i want to use HTTP server stage to achieve the same. Can anyone suggest how to achieve this. Appreciate your help!!Thanks in Advance!!

4 months ago

ajinkyaStreamSets Employee

asked in Show us your Pipelines

Sorting a specific column and writing it to a new table

Use Case:-We have a dataset, in which we have columns as follows:-FIRST_NAME, LAST_NAME, EMAIL, PHONE, GENDER, DEPARTMENT, JOB_TITLE, YEARS_OF_EXPEREIENCE, SALARY. Lets sort the column SALARY in ascending order and write to a new table with just 4 columns, FIRST_NAME, LAST_NAME, YEARS_OF_EXPEREIENCE, SALARY. Pipeline Design:-Snowflake Table (origin) Sort (Processor) Column Remover (Processor) Snowflake Table (Destination) Pipeline Working:-Snowflake Origin will fetch the table and columns and pass the records to Sort processor Sort processor will sort the data based on the configuration and pass it to Column Remover. (SALARY column, Ascending order) Column Remover will keep or remove the columns based on the configuration. Snowflake Table Destination will write the data to a new snowflake table.

4 months ago

AnkurDiscovered Fame

asked in Show us your Pipelines

Error: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

Hi,We need to connect AWS S3 Select using Groovy scripting. For that need to upload jar files. While uploading below Jar files, getting the subject line error. The size of these jars is less than 1MB.joda-time.jarhttpclient.jarhttpcore.jaraws-java-sdk-s3.jarCan you please help me resolving this?Error: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

4 months ago

Anonymous

published in Events & Webinars

New Course: Agile Reporting with StreamSets

We are proud to announce a new, free, and completely self-paced course focused on real world scenarios: Agile Reporting with StreamSets.Learn how to streamline financial reporting through modern, resilient, and repeatable data integration with StreamSets. Quickly and securely integrate and migrate data from internal and external sources, legacy systems, and files into modern data platforms such as Snowflake to accelerate financial reporting.This self-paced hands-on training course teaches you how to overcome several common challenges experienced by organizations attempting to accelerate reporting capabilities:"hard to access" data in legacy systems data silos and data sprawl heavy reliance on IT and a lack of self-service access moving data from legacy to modern data environments integrating internal and external data sources including filesAudienceThe course is designed for data engineers and anyone who is interested in learning how to use StreamSets to solve reporting challenges.Enr

800

5 months ago

newkanyemerch32Fan

asked in Show us your Pipelines

btsmerchstore45

To operate on an EMR cluster, I have developed a straightforward transformer pipeline. I used the required configuration information during cluster configuration.I have the following problem:

6 months ago

tamilarasupDiscovered Fame

asked in Show us your Pipelines

Transformer

In Transformer, i sent data from file(directory) to file(local fs)but the destination will got errorJOBRUNNER_63 - Pipeline status: 'RUN_ERROR', message: 'Operator File_3 failed due to org.apache.spark.sql.AnalysisException, check Driver Logs for further information'RUN_ERROR: Operator File_3 failed due to org.apache.spark.sql.AnalysisException, check Driver Logs for further information (View Stack Trace... ) how to clear the error and I need help to clear the errorThanks Tamilarasu

6 months ago

tamilarasupDiscovered Fame

asked in Show us your Pipelines

Databricks Transformer

Hinow i am working in streamsets data-ops transformer platform. i try to connect databricks cluster in streamsets. But i facing some issues. JOBRUNNER_63 - Pipeline status: 'START_ERROR', message: 'Failed to start pipeline, check logs for error. The Transformer Spark application in the Spark cluster might not be able to reach Transformer at ‘http://661f1312d1e9:19630’. If this is not the correct URL, update the transformer.base.http.url property in the Transformer configuration file or define a cluster callback URL for the pipeline and restart Transformer.'START_ERROR: Failed to start pipeline, check logs for error. The Transformer Spark application in the Spark cluster might not be able to reach Transformer at ‘http://661f1312d1e9:19630’. If this is not the correct URL, update the transformer.base.http.url property in the Transformer configuration file or define a cluster callback URL for the pipeline and restart Transformer. how to clear this error? and how to connect the cluster to

7 months ago

tamilarasupDiscovered Fame

asked in Show us your Pipelines

mongodb

Is possible to perform cdc in mongodb atlas using mongodb oplog? The mongodb oplog connection string will be satisfied for MongoDB atlas but i got time out errorplease give some guidance to clear that error.

7 months ago

Anonymous

published in Events & Webinars

Business Value of Data Engineering Survey is now openNews

The first Business Value of Data Engineering Survey is now open. We know you’re more than a data ‘plumber’ - we see it every day in the outcomes you help your organizations create. With this survey, we’re looking at how you work with your business colleagues to make it happen. Please take 5-7 minutes to fill it out — you’ll automatically be entered to win one of 5 $100 Amazon gift cards! Take the survey here ▶️ https://tinyurl.com/dataengbv

180

8 months ago

PradeepStreamSets Employee

posted in Show us your Pipelines

Example pipelines for Delta lake lookup processor

Input data:[ { "product_id": 1, "unit_price": 23.2 }, { "product_id": 4, "unit_price": 21.64 }, { "product_id": 2, "unit_price": 10.87 }]Delta lake lookup table. This is same for all operations performedReturn all matching rows, generating a record for each match: Return first matching row:There won’t be change in the output from above since there are no duplicates in input. Return a count of matching rows:Return true if matches exist, otherwise false:Input data:[ { "product_id": 1, "unit_price": 23.2 }, { "product_id": 2, "unit_price": 21.64 }, { "product_id": 4, "unit_price": 10.87 }, { "product_id": 5, "unit_price": 99.32 }, { "product_id": 4, "unit_price": 20.33 }]Return all matching rows, generating a record for each match:Return first matching row:Return true if matches exist, otherwise false: Return a count of matching rows:

9 months ago

Santhosh KumarDiscovered Fame

asked in Show us your Pipelines

what is best of streamsets?

Hello Team, My self Santhos kumari'm Beginner of streamsets so which one is best of the learning in depth of streamsets.

11 months ago

lakshmi_narayanan_tDiscovered Fame

asked in Show us your Pipelines

JDBC PIPE LINE NOT RUNING,SHOWS ME ERROR LIKE: JDBC_16 - Table '' does not exist or PDB is incorrect. Make sure the correct PDB was specified

I facing a issue on running JDBC pipeline ,it shows the below error ,I tried so many things from the google but no use anybody can solve .

1 year ago

Page 1 / 4

Badge winners

Sanjeevhas earned the badge Eager to help
AkshayJadhavhas earned the badge Eager to help
john.durkinhas earned the badge Eager to help
samhas earned the badge Eager to help
saleempothiwalahas earned the badge Eager to help

Show all badges

Terms & Conditions

Sign up

Already have an account? Login

Social Login

Username *

E-mail address *

What I do... *

Data Leader Data Architect Data Engineer Data Scientist Other

Company *

Country *

Zip Code *

Marketing Communications

Yes No

Password *

I have read and Agree to the Website Terms of Service and I have read and acknowledged the Privacy Policy.

loginBox.register.email_repeat

Login to the community

No account yet? Create an account

Social Login

Username or Email

Password

Remember me

Forgot password?

Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.

Username or e-mail

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.