- - Knowledge base
Product Updates
Events

30-Day Free Trial: It’s Never Been Easier To Get Started With StreamSets

4 months ago

Home
Community overview
StreamSets Platform

StreamSets Platform

Get inspired and gain all the knowledge you need

1088 Topics

Recently active

Most replies Most views Newest first

john.mcavoyStreamSets Employee

asked in Community Articles and Got a Question?

BikramHeadliner

asked in Community Articles and Got a Question?

How to remove special characters from field names in streamsets data collector?

Solution.The issue can be handled by Field mapper processor. Below the syntax for the same. ${str:replaceAll(f:name(), '[.\\-/()!&%^$#@]','')} Attached the screen shot for the reference. NB : The issue also can be handled in different ways e;g(Expression Evaluator, scripting processor etc)

5 days ago

AkshayJadhavStreamSets Employee

posted in Share your Best Practices

Kafka Multi-topic consumer failing with "IndexOutOfBoundsException" when produce single message is true.

Environment:Streamsets Data Collector v4.2.1. Kafka Multi-topic consumer origin. Issue:It has been observed that Kafka origin is failing with the following error message when “produce single records” configuration is on + data is in JSON format.2021-01-10 15:31:12,193 [user:test] [pipeline:test_kafka_working] [runner:] [thread:Kafka pipeline-thread1] [stage:KafkaMultitopicConsumer_01] INFO MultiKafkaSource - Multi kafka thread halted unexpectedly: KAFKA_29 - Error fetching data from Kafka: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0java.util.concurrent.ExecutionException: com.streamsets.pipeline.api.StageException: KAFKA_29 - Error fetching data from Kafka: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.streamsets.pipeline.stage.origin.multikafka.MultiKafkaSource.produce(MultiKafkaSource.java:163) at com.streamsets.pipeline.ap

6 days ago

sudarshan_shinde1Fan

asked in Community Articles and Got a Question?

Export external jar using python sdk.

Hi,We need to export jar files/libraries of specific engine of one control hub to another control hub using Python SDK. How can we achieve it? Thanks.

9 days ago

simba01Fan

asked in Community Articles and Got a Question?

Why does the Speakatoo API function properly in POSTMAN but not within my integration?

The Speakatoo API functions correctly in POSTMAN, but it isn't working as expected in my integration for Text To Speech conversion. What might be the issue?

10 days ago

meowwFan

asked in Community Articles and Got a Question?

S3 pipelines on SDC v5.6.0 failing with AmazonS3Exception:SignatureDoesNotMatch after upgrading to v5.6.0 from v5.5.0

we upgraded data collector to v5.6.0 from v5.5.0 recently… but since then getting our s3 pipelines are failing with below ERROR.. we are using `Instance Profile` as authentication method..Also data collector with v5.5.0 which are deployed same (AWS acc)environment are working fines with s3 pipelines & instance_profile authentication method. anyone else faced similar kind of problem?RETRY: S3_21 - Unable to write object to Amazon S3, reason : com.amazonaws.services.s3.model.AmazonS3Exception: The request signature we calculated does not match the signature you provided. Check your key and signing method. (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: 3VFHG02TG4GR3KTR; S3 Extended Request ID: PePeTM0p51ZFcECJ/Mw6PU9nK4Km6AWNEaWhcykqj2ovDW/IbGcMtcaK+xcRxlMoPuRlCyxh2vo=; Proxy: null), S3 Extended Request ID: PePeTM0p51ZFcECJ/Mw6PU9nK4Km6AWNEaWhcykqj2ovDW/IbGcMtcaK+xcRxlMoPuRlCyxh2vo=

11 days ago

lakshmanFan

asked in Community Articles and Got a Question?

Reading CSV file it shows extra record .

My csv file has only Two Record But it shows ,extra records in stream sets Environment.this is my csv record data but Building Information column had a paragraph data.In stream sets 5 records Showing .anyone can suggest how to handle this file.

12 days ago

drozdse1Roadie

asked in Community Articles and Got a Question?

Set data type of a new added field (Expression Evaluator)

How can I change the type of a new added field in the Expression Evaluator? Because if I add a field like this: Then the field is defined as a string. But I need to have it as DATETIMEI know I could just not add this field but my goal is to make the pipeline creating not existing tables if needed. So I need to declare all attributes, even they are empty. Thanks

12 days ago

somFan

posted in Community Articles and Got a Question?

Include older sdc logs(sdc.log & sdc-1.log extension) in Support Bundle

Previously on all data collector below 5.X version , The behaviour was that all SDC logs were included in the Support Bundle. Now only the current log(sdc.log) is included and sdc-*.log logs are missing.The solution for the fix was applied post 5.4 version onwards.

12 days ago

somFan

posted in Community Articles and Got a Question?

View list of jars that has been loaded during SDC start up

We can see the list of jars that has been loaded during SDC start up by below configuration,If using Cloudera Manager, Please add the following to the Data Collector Advanced Configuration Snippet (Safety Valve) for sdc-env.sh configuration and restart SDC: export SDC_JAVA_OPTS="-Dstreamsets.bootstrap.debug=true ${SDC_JAVA_OPTS}" This will tell us what jars are being picked up in the classpath when SDC starts. The output will appear in stdout.log of the Data Collector process. If we are using standalone SDC, we can add the above snippet directly to sdc-env.sh ($SDC_HOME/libexec)and restart the sdc.

12 days ago

drozdse1Roadie

asked in Community Articles and Got a Question?

DATE type in Oracle origin database is casted into DATETIME on Snowflake?

Our goal is to sync between Oracle and Snowflake with CDC Oracle. Some of our source tables in Oracle are defined as DATE. So we created this tables also in Snowflake with the same type, DATE. But now I get this error message: But in this case the field ACTIVITY_TIME is DATE on Oracle and Snowflake.Later I noticed that we don’t have any problem if I convert all DATE fields into DATETIME attributes on Snowflake. But looking in the documentation, there is nothing mentioned that we have to use DATETIME in such cases. The type DATE is explicitly mentioned as supported type. Does anyone had the same issues? I mean it’s not so unusual to use the DATE type in a database

13 days ago

catchcharanFan

asked in Community Articles and Got a Question?

HL7 format

Hi Team, Does StreamSets support HL7 data format? I know there is a python way of doing this. Thanks,Meher

17 days ago

ajinkyaStreamSets Employee

posted in StreamSets Academy

MONGODB_22 - Mongo Query Error: Command failed with error 115 (CommandNotSupported): 'command insert is unsupported

Issue:- While connecting to the MongoDB atlas, following error has been occurred:-MONGODB_22 - Mongo Query Error: Command failed with error 115 (CommandNotSupported): 'command insert is unsupported, correlationID = 179543fa79a5cba7e589f5b1' on server atlas-sql-653f56d6e2904d1155c97d0e-e54eg.a.query.mongodb.net:27017. The full response is {"ok": 0, "errmsg": "command insert is unsupported, correlationID = 179543fa79a5cba7e589f5b1", "code": 115, "codeName": "CommandNotSupported"} null Solution:-The query is failing due to the authentication issue. We have to provide a Username and Password in the JDBC string for the MongoDB Atlas.mongodb+srv://<USERNAME>:<PASSWORD>@<MONGODB_ATLAS_URL>/?retryWrites=true&w=majority

19 days ago

ajinkyaStreamSets Employee

asked in Share your Best Practices

MONGODB_13 - Error serializing record: java.lang.UnsupportedOperationException: BSON Converter cannot convert java.util.ArrayList to BSON Document

Issue:-While adding a json data to the MongoDB collection, following error has been observed.MONGODB_13 - Error serializing record: java.lang.UnsupportedOperationException: BSON Converter cannot convert java.util.ArrayList to BSON DocumentSolution:- MongoDB atlas was reading the record as a list. As the JSON data was enclosed with the [].We have to set the JSON Content to JSON array of objects at the Origin side.

19 days ago

MalathiFan

asked in Community Articles and Got a Question?

SDC Snowflake destination is not connecting

Hi,I am trying to load data into snowflake in SDC. But, I am getting below error. Can you please tell me where I am doing wrong because it is working fine in Transfomer for snowflake for the same snowflake account url. In SDC it is not working if I am using the locator on account tab. SNOWFLAKE_11 - Could not create SQL DataSource: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: JDBC driver encountered communication error. Message: Exception encountered for HTTP request: Certificate for <kxxxxxx.eu-north-1.snowflakecomputing.com> doesn't match any of the subject alternative names: [*.prod3.us-west-2.snowflakecomputing.com, *.us-west-2.snowflakecomputing.com, *.global.snowflakecomputing.com, *.snowflakecomputing.com, *.prod3.us-west-2.aws.snowflakecomputing.com]. Thanks,Malathi

20 days ago

lakshmi_narayanan_tDiscovered Fame

asked in StreamSets Academy

how to covert file and write into localfs as a XLSX file Format.

can you please suggest any other configuration or proseccor to achieve output.i reading my file in directory origin origin is :configuration my destination is localfs:configuration after execution of my pipeline :-out put file like - But my requirement is like how to acive my fie format ,by using any processor jython evaluvator or whole file transformer.

20 days ago

Dhanashri_BhateOpening Band

asked in Community Articles and Got a Question?

GCP: credential:get function - Are there limitations where this function can be used?

Hello, One of my customer is using StresmSets to access secrets saved in GCP. The SDC setup and GCP secrets setup is correct. For example, I have used secrets like below successfully in JDBC producer stage’s credential tab and pipeline works fine, picks up latest version of the secret as expected. ${credential:get("gcp", "group@org", "dbuser?latest")}${credential:get("gcp", "group@org", "dbpass?latest")}Now, they want to read a secret for which a json file was uploaded as value, and they want to extract credential information from this json. I tried to use credential:get in an Expression evaluator but it gave an error. (was planning to add more processors after this to extract userid and password from the json returned by credential:get) How can i use secret like this in my StreamSets pipeline? Thanks!

20 days ago

ccarimanFan

asked in StreamSets Academy

How StreamSets Stop or kill jobs running on Data Collector engine?

Would you like to share How StreamSets Stop or kill jobs running on Data Collector engine? (Control Hub)I ask this because I cannot access the job from the engine detail, in the "Running Pipelines" section. When trying to access the job instance I get the following error: "HTTP Status: 404 (Not Found): " maybe kill the job from command line? regards. Carlos.

22 days ago

Mike CalmaStreamSets Employee

posted in Community Articles and Got a Question?

ADLS Gen 2 Legacy

[ADLS Gen 2 Changes]:In Data Collector Version 5.5.0 a change was made which altered how the ADLS Gen 2 stage functioned. Previously the ADLS Gen 2 Stage utilized the Hadoop FS to access resources. This change also split the stage into the respective resources they access: ADLS Gen 2 and Azure Blob Storage. For additional information surrounding this change please review the patch notes linked below: Ref 1: Patch NotesRef 2: Available Stage Comparison [Legacy ADLS Gen 2]:Additionally, the ADLS Gen 2 Legacy stage which used Hadoop FS is also still available to be utilized.** It should be noted though that when exporting a pipeline made on a version prior to 5.5 into later Data Collectors that the (Legacy) tag will not be present on the stage information. However, the description of the stage will still relay the method with which the resource is accessed. When the (Legacy) stage is selected in a current pipeline it can be identified as seen in the image below

24 days ago

blakeorsakFan

posted in Community Articles and Got a Question?

Excessively large engine-$$.out file in the /tmp directory

The Issue: Data Collector has created a file in the /tmp directory that is growing to an excessive size and is taking up space. The purpose of this file is to serve as a location for stdout from a running Data Collector. Under normal circumstances, this file does not receive much data and should not grow to a large size. Specifically, the rotatelogs is used to limit the size of this file to ~10mb. However, there are a few instances where this file will receive more data than usual, or grow to an excessively large size. Adding certain Java parameters to your $SDC_HOME/libexec/sdc-env.sh file can result in extra output being sent to this /tmp file. This output is often useful for debugging, but can fill the file up quickly. An example of this might be the -Djavax.net.debug=ssl,handshake parameter used to debug SSL connections. If the rotatelogs utility is not currently available on your machine, the /tmp/engine-$$.out file may continue to grow past the limited size of ~10mb. The So

24 days ago

SindhuStreamSets Employee

posted in Share your Best Practices

Pipeline ID change.

Does the pipeline ID change, when any changes are made to the pipeline and re-published ?Pipeline ID is assigned during pipeline creation and does not change under any circumstances.

24 days ago

FranciscoFan

asked in Community Articles and Got a Question?

HTTP Processor for POST request clearing header values

Hello! I’m using record header attributes as variables. For example, during pipeline execution, saving ${time:now()} in a header attribute with an Expression Evaluator, to be used in other stages. This is no problem for HTTP GET requests, as the data gets aggregated into existing records. The problem is with HTTP Processor for POST requests, where the response overwrites all existing data from before, body and headers.Is there anyway around this? I’m trying to use this header attribute value that I saved to be used in anohter HTTP POST request after the one I just mentioned, but the value is lost. Thanks in advance!

25 days ago

AkshayJadhavStreamSets Employee

posted in Share your Best Practices

Configuring Google Secret Manager Credential Store for data collector.

In this article, we will walk you through the necessary steps to configure and set up Google Secrets for use with Data Collector.Firstly, We have to create the secret on the GCP which we will be using in the pipeline. In this article, I am going to store the mysql username and password in secrets.These are the secret created in GCP:- The pre-requisite is to install the package Google Secret Manager Credential Store on the Data collector and then update the following properties in credential-stores.properties file.credentialStores=gcp credentialStore.gcp.config.project.id= <gcp-projectID> credentialStore.gcp.config.credentialsMode=jsonPath credentialStore.gcp.config.credentialsJsonPath=/Users/apple/.config/gcloud/application_default_credentials.jsonOnce done, restart the data collector. This implicit EL expression we are going to use in the pipeline credentials pipeline:Username => ${credential:get("gcp", "all@all", "akshay_sqlusername?latest")}Password => ${credential:get("

1 month ago

Page 1 / 44

Badge winners

Sanjeevhas earned the badge Eager to help
AkshayJadhavhas earned the badge Eager to help
john.durkinhas earned the badge Eager to help
samhas earned the badge Eager to help
saleempothiwalahas earned the badge Eager to help

Show all badges

Terms & Conditions

Sign up

Already have an account? Login

Social Login

Username *

E-mail address *

What I do... *

Data Leader Data Architect Data Engineer Data Scientist Other

Company *

Country *

Zip Code *

Marketing Communications

Yes No

Password *

I have read and Agree to the Website Terms of Service and I have read and acknowledged the Privacy Policy.

loginBox.register.email_repeat

Login to the community

No account yet? Create an account

Social Login

Username or Email

Password

Remember me

Forgot password?

Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.

Username or e-mail

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.