- - Knowledge base
Product Updates
Events

30-Day Free Trial: It’s Never Been Easier To Get Started With StreamSets

1 year ago

Home
Community overview
StreamSets Platform

StreamSets Platform

Get inspired and gain all the knowledge you need

1200 Topics

Recently active

Most replies Most views Newest first

ufmurphyOpening Band

asked in StreamSets Academy

DataOps Platform Fundamentals Course - Data Drift missing

Hey guys, In the DataOps Platform Fundamentals Course, the video on Data Drift is missing.https://academy.streamsets.com/courses/dataops-platform-fundamentals/lessons/data-drift-2/ Same with CI/CD:https://academy.streamsets.com/courses/dataops-platform-fundamentals/lessons/implementing-pipeline-ci-cd/ Is there a video, or should I mark it complete and move on? Jim

723

2 years ago

RMOFan

asked in Community Articles and Got a Question?

HDFS KMS provider at [http://x.y.z:16000/kms/v1/] threw an IOException

Hello,We are getting an error in Hadoop FS HDFS destination while writing a file. It works fine when we use the Stage Library version CDH 5.16 but fails if we use any higher version like CDH 6.x or CDP 7.1. What may be causing this if the same setup works in SDC using CDH5.16 stage library. Also it works fine for version CDP 7.1 if we do not use HDFS encrypted directory since it wont call KMS provider.The error in the log states: WARN KMS provider at [http://x.y.z:16000/kms/v1/] threw an IOException:java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:487) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:788)

2530

2 years ago

ufmurphyOpening Band

asked in StreamSets Academy

Runtime parameters: reset origin and start

Hey Community, I see the option in the Streamsets canvas:Start pipeline Reset origin and start Start with parameters Is it possible to combine the last 2 options?Reset origin and start with parameters

5153

2 years ago

Ansul141185Fan

asked in Community Articles and Got a Question?

sybase ase jdbc connection strings in jdbc producer

When I am using jdbc producer to insert data into Sybase ASE its throwing me an error : JDBC_17 -failed to lookup primary keys for table ‘xxxxxx’ : SQLState: ZZZZZ Error code:2762 Message: The ‘CREATE TABLE’ command is not allowed within a multi-statement transaction in the ‘tempdb’ database. SQLState: 42000 Error code:208 Message:#keys not found.’ Please have a look and advice.

1131

2 years ago

ronaldmFan

asked in Community Articles and Got a Question?

How to perform scp in Data Collector

Is it possible to perform a whole file scp using the Data Collector?

482

2 years ago

balajiitFan

asked in Community Articles and Got a Question?

ask.streamsets.com contents lost?

There are plenty of Q/A found under ask.streamsets.com and I can’t find them here. Is there a way I can search for those Q/As?

571

2 years ago

akkisabbyFan

asked in Community Articles and Got a Question?

Compare records from 2 sources in Jython Evaluator

I am trying to use Jython Evaluator for fuzzy matching. I need to compare a record from one source with another in Jython code and emit only 1 record from evaluator.So my question is how do i access records (values) from 2 different sources at the same in Jython evaluator?Request for assistance

1452

2 years ago

jasonFan

asked in Community Articles and Got a Question?

Prevent RabbitMQ Producer from creating queue with exchange

Hello. In one of our pipelines, we end with a call to a Rabbit MQ Producer node. The node publishes to a fanout exchange. Later in another application, we create a listener queue on that same exchange and listen for messages on the newly created queue. This happens for each docker container running our application which is listening on the fanout exchange. As a side effect, of the Rabbit MQ Producer node creating the exchange, it is also creating a queue along with it. This results in a queue being sent all the messages that are sent to the exchange which has no consumers. We would like to prevent this default queue from being created. Is there a way to do that in the RabbitMQ Producer node? Below I will outline the steps to try and make the situation more clearRabbit MQ Producer node creates fanout Exchange Rabbit MQ Producer node creates queue “A” on the exchange Instance 1 of separate application creates queue “B” on the exchange Instance 2 of separate application creates qu

1604

2 years ago

jangala.karthikFan

asked in Community Articles and Got a Question?

More than one condition in stream selector

In the stream selector, can we specify more than on condition in one stream? I have one more question not related to stream selector. How can we add line breaks in the email body field in the email executor?

7645

2 years ago

ufmurphyOpening Band

asked in StreamSets Academy

Kafka not running in lab

Hey guys, I am running through the Academy training and seem to have hit a snag. I am not real knowledgeable about Docker/Kubernetes, so I thought I’d ask here. Is Kafka running in the labs? I am getting timeout errors when I try to connect from Control Hub.https://academy.streamsets.com/courses/dataops-platform-fundamentals/lessons/build-a-kafka-pipeline/topic/lab-build-a-kafka-pipeline-2/ Any help is appreciated. Jim

802

2 years ago

Srinivasan SankarHeadliner

asked in Community Articles and Got a Question?

Pipeline State History JSON file

I am on Data Collector 3.21.x. I noticed that the “PipelineStateHistory.json” file keeps growing in size. Are there are workarounds or suggestions to just keep the last few days or last few runs history in-order to limit the file size growth?

1552

2 years ago

DashSenior Technical Evangelist and Developer Advocate at Snowflake

asked in Community Articles and Got a Question?

What is StreamSets Summer ‘21?

Hi! I’d like to know more about StreamSets Summer ‘21.

1063

2 years ago

uacateFan

asked in Community Articles and Got a Question?

SFTP Pipeline, Job, and Scheduler

I have a streamsets pipeline that part of a scheduled job. The pipeline reads a CSV file that is stored at an AWS sftp location. That csv file gets overwritten every night. The scheduled job is supposed to read the file well after the file is over-written. The scheduler does run the job at the specified hour, however, the pipeline only reads the first line of the csv file and ends even though there are many records for it to process. If I manually run the job and reset the origin the job runs as expected. I have only been working with streamsets for about 5 months. Anyone suggest what I might be missing in my pipeline or possibly the scheduler?

1904

2 years ago

Srinivasan SankarHeadliner

posted in Community Articles and Got a Question?

Data Collector - Field Renamer stage

I am on Data Collector 3.21.xI observed the following and I would like to share with the community. Unsure whether this is a bug OR a gap that needs to be addressed in the documentation. My pipeline is simple: Reads from a SQL Sever DB table and writes to AWS S3.I use a Origin stage followed by a “Field Renamer” stage. Scenario 1: When I use the JDBC multi-table consumer origin, i.e., I read all the fields from the source table I do not need to use the “REPLACE” option in “Rename” tab within the “Field Renamer” stage. Refer picture below, Pipeline Design Field Renamer Stage Config. Scenario 2:When I use the JDBC Query Consumer stage as my origin and I issue a SELECT statement that is NOT SELECT * FROM …. i.e., I only select a few columns. Example: Select <column1> from <table1>, I have to use the REPLACE option in “Target Field Already Exists” otherwise my pipeline fails. Pipeline Design com.streamsets.pipeline.api.base.OnRecordErrorException: FIELD_RENAMER_01 - Target

3040

2 years ago

Srinivasan SankarHeadliner

asked in Community Articles and Got a Question?

Data Collector RESTful API - GET operation on Pipeline Metrics

Hi there, I am on Data Collector 3.21.x. I tried issuing a RESTFul API call for one of our pipeline to get the metrics for a pipeline. But I get the following response, My Request URL is,/rest/v1/pipeline/snowflake432db0d5-22d5-4745-b7da-55bf1db2ee16/metrics?rev=0 My model schema is,{ "version": "string", "gauges": {}, "counters": {}, "histograms": {}, "meters": {}, "timers": {}, "slaves": [ "string" ]} Any suggestions?

2932

2 years ago

denisaungureanuFan

asked in Community Articles and Got a Question?

Send Data Sequentially

Hello everyone, I have a csv file and I want to send data to a REST API.My final data looks like this:I will give you an example of how I want my final result to be.For 3 records in my csv file, every record will look similar with the one above. For 1 record in my csv file, I want to sent firstly a json with serial, interface and userData_dep fields, and then another json with serial, interface and userData_lan fields.To do that I used 2 http client processors, one for the dep and one for the lan. But I saw that the jsons weren’t sent as I want. My pipeline send 3 json with _dep field, and then another 3 json with _lan field.I want to send 1 json with dep field, then 1 json with lan field, then the next json with the dep field, and the next json with the lan field and so on.I want to send them sequentially.Could you help me with some ideas?Thank you!

330

2 years ago

anirbanchFan

asked in Community Articles and Got a Question?

StreamSets Data Collector with Golden Gate as Oracle CDC

We understand the Oracle CDC client for CDC from Oracle using SDC. But does SDC support Golden Gate as CDC origin please?

691

2 years ago

Srinivasan SankarHeadliner

asked in Community Articles and Got a Question?

STAGE ERROR: Data Collector pipeline loading into Snowflake DB

I have a data collector pipeline that runs on data collector version 3.21.xThis pipeline reads data using a JDBC Multi-table Consumer Origin and writes to a Snowflake DB/table. The pipeline used to run successfully. But in the past two days we encounter the following error, "errorMessage": "SNOWFLAKE_59 - Could not get a Stage File Writer instance to write the records: 'SNOWFLAKE_65 - File upload to Snowflake failed: net.snowflake.client.jdbc.SnowflakeSQLLoggedException: S3 operation failed: Operation=listObjects, Error type=Client, Error code=AccessDenied, Error message=Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: DK041FE70QRW4B85; S3 Extended Request ID: DCSX2ikBaa3k/Vwcsc5+Cbb+pH+7KkLITtJYr+e3ea7cHlOVvVeDG9xhpZtQnbc/6FrAsef4C/w=), Request id=DK041FE70QRW4B85, Extended request id=DCSX2ikBaa3k/Vwcsc5+Cbb+pH+7KkLITtJYr+e3ea7cHlOVvVeDG9xhpZtQnbc/6FrAsef4C/w='", Note: I am using the Snowflake Internal Stage NOT AWS S3 and hence unsure what pe

6446

2 years ago

BikramHeadliner

asked in Community Articles and Got a Question?

How to send files to sftp/mft servers

In order to send the file from source system to SFTP/MFT servers , we need to select the data format as “Whole file” in the origin because the sftp processor only accept the data format as whole file.

750

3 years ago

swayamDiscovered Fame

posted in Community Articles and Got a Question?

Building a No-code Data Connector Platform Using StreamSets Data Collector REST APIs

PrefaceFetching online and offline data across different channels to understand your end-users behavior is a primary topic for any enterprise-level brand. Customer 360 view of an end customer gives enormous insight to marketing leaders who can then prepare their strategy to engage and entice these users. A full-stack CDP (Customer Data Platform) helps meet the needs of marketing leaders and analytic folks.A stable, reliable, and scalable data processing stack is the de facto need for any enterprise CDP solution, and to set up that, data ingestion and extraction architecture need to be solid. StreamSets Data Collector (SDC) has become the preferred tool for many data engineers and data stewards to achieve the ETL and ELT-related needs in this area.In the traditional approach, the ETL engineers normally have definitive requirements in a place where they know the origin, transformation rules, and destination. Based on the requirements, engineers develop their pipelines and deploy them to

3090

3 years ago

mtakaFan

asked in Community Articles and Got a Question?

Convert String to Datetime Error

Hello,I am using a Field Type Converter to convert string to datetime with date format (yyyy-MM-dd'T'HH:mm:ss.SSS'Z' - 2021-08-18T08:20:24.136000Z). However, the same field has dates with format 2021-08-18T08:20:24.00Z, which it gives me error CONVERTER_00 - Failed to convert field '/createdate' of type 'STRING' with value '2021-08-19T14:01:32Z' to type 'DATETIME' Any recommendations on how to handle two different date formats for the same field?

14636

3 years ago

javierdallamoreFan

posted in Community Articles and Got a Question?

Our Journey Finding the Perfect-Fit Data Integration Tool

Overview Since November 2014 we have been implementing a Data Lake platform to collect, process, filter, enrich, store, analyse and visualize logs and business events using Big Data and streaming technologies. The main goal is to detect important events, generate statistics and produce alerts to solve problems in a proactive way.Our first pilot on a new customer is usually related to application logs but once this platform is up and running it naturally becomes the centralized repository not only for logs but also for any type of sources and data formats. With dashboards on display in big screens other areas and departments start requesting dashboards of their own that require new data sources to be connected.To adapt to different use cases and new requirements we needed a more modular, flexible and extensible architecture. As the Data Lake gained prominence within our firsts customers and other project opportunities materialized, challenges and constraints emerged that led us to rethi

1280

3 years ago

SebastienMRoadie

asked in StreamSets Academy

Getting error: VALIDATION_0011 - Stage has open output streams

I get this error, following the steps in the lecture “DataOps Platform Fundamentals”, chapter “Build a Pipeline”, lab “Lab: Build a Pipeline”.When clicking on the error, it takes me to the Stream Selector module, Configuration, General tab. There is nothing highlighted in that section and all seems in working order to me.Could you please advise?

51611

3 years ago

Diana LacatusFan

asked in Community Articles and Got a Question?

JSON Object

Hello I tried to extract and transform data from an CSV file and to obtain as an output a JSON object. I used the JSON generator and in the preview area things looked good. I will attach here the output.Output from DCI tried to sent this object to a REST API but when I looked into the logs, I realized that the format of the input was not the same as I expected. Some backslashes are added before every field and every value.Input from logsCould you please help me with some ideas or maybe a solution to my problem? Thank you,Diana

1702

3 years ago

wasifkabeerFan

asked in Community Articles and Got a Question?

Cloud Anti-virus Scan via ICAP in Streamsets DC

Dear Experts, I have a simple SFTP data pipeline to take text files from an SFTP cloud server and push these files to a local file system.However, I am required to get these files scanned by an anti-virus via ICAP call "icap://${SERVER_IP}:${SERVER_PORT}/virusScan" before writing to the target.Can anyone guide me on how to get this done in Streamsets?FYI: some other tools like Informatica have a built-in option for this. Please refer this link:https://knowledge.informatica.com/s/article/613287?language=en_US&type=external

312

3 years ago

Page 46 / 48

Badge winners

Ykyeeywkkwhas earned the badge Eager to help
vishwesh.margasahayamhas earned the badge Product expert
ajinkyahas earned the badge Innovator
Sanjeevhas earned the badge Eager to help
AkshayJadhavhas earned the badge Eager to help

Show all badges

Terms & Conditions

Sign up

Already have an account? Login

Social Login

Username *

E-mail address *

What I do... *

Data Leader Data Architect Data Engineer Data Scientist Other

Company *

Country *

Zip Code *

Marketing Communications

Yes No

Password *

I have read and Agree to the Website Terms of Service and I have read and acknowledged the Privacy Policy.

loginBox.register.email_repeat

Login to the community

No account yet? Create an account

Social Login

Username or Email

Password

Remember me

Forgot password?

Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.

Enter your e-mail address

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.