- - Knowledge base
Product Updates
Events

30-Day Free Trial: It’s Never Been Easier To Get Started With StreamSets

4 months ago

Home
Community overview
StreamSets Platform

StreamSets Platform

Get inspired and gain all the knowledge you need

1086 Topics

Recently active

Most replies Most views Newest first

projectpollonetworthFan

asked in Community Articles and Got a Question?

Can we see who answered the question in a live poll?

Good day,Would it be possible to get the live replies to a poll using an account? Or is exporting replies the sole method to view them once the poll is over? (and which accounts permit this, if any) I am grateful,

23 hours ago

BikramHeadliner

asked in Community Articles and Got a Question?

How to remove special characters from field names in streamsets data collector?

Solution.The issue can be handled by Field mapper processor. Below the syntax for the same. ${str:replaceAll(f:name(), '[.\\-/()!&%^$#@]','')} Attached the screen shot for the reference. NB : The issue also can be handled in different ways e;g(Expression Evaluator, scripting processor etc)

1 day ago

AkshayJadhavStreamSets Employee

posted in Share your Best Practices

Kafka Multi-topic consumer failing with "IndexOutOfBoundsException" when produce single message is true.

Environment:Streamsets Data Collector v4.2.1. Kafka Multi-topic consumer origin. Issue:It has been observed that Kafka origin is failing with the following error message when “produce single records” configuration is on + data is in JSON format.2021-01-10 15:31:12,193 [user:test] [pipeline:test_kafka_working] [runner:] [thread:Kafka pipeline-thread1] [stage:KafkaMultitopicConsumer_01] INFO MultiKafkaSource - Multi kafka thread halted unexpectedly: KAFKA_29 - Error fetching data from Kafka: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0java.util.concurrent.ExecutionException: com.streamsets.pipeline.api.StageException: KAFKA_29 - Error fetching data from Kafka: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.streamsets.pipeline.stage.origin.multikafka.MultiKafkaSource.produce(MultiKafkaSource.java:163) at com.streamsets.pipeline.ap

2 days ago

sudarshan_shinde1Fan

asked in Community Articles and Got a Question?

Export external jar using python sdk.

Hi,We need to export jar files/libraries of specific engine of one control hub to another control hub using Python SDK. How can we achieve it? Thanks.

5 days ago

simba01Fan

asked in Community Articles and Got a Question?

Why does the Speakatoo API function properly in POSTMAN but not within my integration?

The Speakatoo API functions correctly in POSTMAN, but it isn't working as expected in my integration for Text To Speech conversion. What might be the issue?

6 days ago

meowwFan

asked in Community Articles and Got a Question?

S3 pipelines on SDC v5.6.0 failing with AmazonS3Exception:SignatureDoesNotMatch after upgrading to v5.6.0 from v5.5.0

we upgraded data collector to v5.6.0 from v5.5.0 recently… but since then getting our s3 pipelines are failing with below ERROR.. we are using `Instance Profile` as authentication method..Also data collector with v5.5.0 which are deployed same (AWS acc)environment are working fines with s3 pipelines & instance_profile authentication method. anyone else faced similar kind of problem?RETRY: S3_21 - Unable to write object to Amazon S3, reason : com.amazonaws.services.s3.model.AmazonS3Exception: The request signature we calculated does not match the signature you provided. Check your key and signing method. (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: 3VFHG02TG4GR3KTR; S3 Extended Request ID: PePeTM0p51ZFcECJ/Mw6PU9nK4Km6AWNEaWhcykqj2ovDW/IbGcMtcaK+xcRxlMoPuRlCyxh2vo=; Proxy: null), S3 Extended Request ID: PePeTM0p51ZFcECJ/Mw6PU9nK4Km6AWNEaWhcykqj2ovDW/IbGcMtcaK+xcRxlMoPuRlCyxh2vo=

6 days ago

lakshmanFan

asked in Community Articles and Got a Question?

ajinkyaStreamSets Employee

posted in StreamSets Academy

MONGODB_22 - Mongo Query Error: Command failed with error 115 (CommandNotSupported): 'command insert is unsupported

Issue:- While connecting to the MongoDB atlas, following error has been occurred:-MONGODB_22 - Mongo Query Error: Command failed with error 115 (CommandNotSupported): 'command insert is unsupported, correlationID = 179543fa79a5cba7e589f5b1' on server atlas-sql-653f56d6e2904d1155c97d0e-e54eg.a.query.mongodb.net:27017. The full response is {"ok": 0, "errmsg": "command insert is unsupported, correlationID = 179543fa79a5cba7e589f5b1", "code": 115, "codeName": "CommandNotSupported"} null Solution:-The query is failing due to the authentication issue. We have to provide a Username and Password in the JDBC string for the MongoDB Atlas.mongodb+srv://<USERNAME>:<PASSWORD>@<MONGODB_ATLAS_URL>/?retryWrites=true&w=majority

14 days ago

ajinkyaStreamSets Employee

asked in Share your Best Practices

MONGODB_13 - Error serializing record: java.lang.UnsupportedOperationException: BSON Converter cannot convert java.util.ArrayList to BSON Document

Issue:-While adding a json data to the MongoDB collection, following error has been observed.MONGODB_13 - Error serializing record: java.lang.UnsupportedOperationException: BSON Converter cannot convert java.util.ArrayList to BSON DocumentSolution:- MongoDB atlas was reading the record as a list. As the JSON data was enclosed with the [].We have to set the JSON Content to JSON array of objects at the Origin side.

14 days ago

MalathiFan

asked in Community Articles and Got a Question?

SDC Snowflake destination is not connecting

Hi,I am trying to load data into snowflake in SDC. But, I am getting below error. Can you please tell me where I am doing wrong because it is working fine in Transfomer for snowflake for the same snowflake account url. In SDC it is not working if I am using the locator on account tab. SNOWFLAKE_11 - Could not create SQL DataSource: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: JDBC driver encountered communication error. Message: Exception encountered for HTTP request: Certificate for <kxxxxxx.eu-north-1.snowflakecomputing.com> doesn't match any of the subject alternative names: [*.prod3.us-west-2.snowflakecomputing.com, *.us-west-2.snowflakecomputing.com, *.global.snowflakecomputing.com, *.snowflakecomputing.com, *.prod3.us-west-2.aws.snowflakecomputing.com]. Thanks,Malathi

15 days ago

lakshmi_narayanan_tDiscovered Fame

asked in StreamSets Academy

how to covert file and write into localfs as a XLSX file Format.

can you please suggest any other configuration or proseccor to achieve output.i reading my file in directory origin origin is :configuration my destination is localfs:configuration after execution of my pipeline :-out put file like - But my requirement is like how to acive my fie format ,by using any processor jython evaluvator or whole file transformer.

15 days ago

Dhanashri_BhateOpening Band

asked in Community Articles and Got a Question?

GCP: credential:get function - Are there limitations where this function can be used?

Hello, One of my customer is using StresmSets to access secrets saved in GCP. The SDC setup and GCP secrets setup is correct. For example, I have used secrets like below successfully in JDBC producer stage’s credential tab and pipeline works fine, picks up latest version of the secret as expected. ${credential:get("gcp", "group@org", "dbuser?latest")}${credential:get("gcp", "group@org", "dbpass?latest")}Now, they want to read a secret for which a json file was uploaded as value, and they want to extract credential information from this json. I tried to use credential:get in an Expression evaluator but it gave an error. (was planning to add more processors after this to extract userid and password from the json returned by credential:get) How can i use secret like this in my StreamSets pipeline? Thanks!

15 days ago

ccarimanFan

asked in StreamSets Academy

How StreamSets Stop or kill jobs running on Data Collector engine?

Would you like to share How StreamSets Stop or kill jobs running on Data Collector engine? (Control Hub)I ask this because I cannot access the job from the engine detail, in the "Running Pipelines" section. When trying to access the job instance I get the following error: "HTTP Status: 404 (Not Found): " maybe kill the job from command line? regards. Carlos.

18 days ago

Mike CalmaStreamSets Employee

posted in Community Articles and Got a Question?

ADLS Gen 2 Legacy

[ADLS Gen 2 Changes]:In Data Collector Version 5.5.0 a change was made which altered how the ADLS Gen 2 stage functioned. Previously the ADLS Gen 2 Stage utilized the Hadoop FS to access resources. This change also split the stage into the respective resources they access: ADLS Gen 2 and Azure Blob Storage. For additional information surrounding this change please review the patch notes linked below: Ref 1: Patch NotesRef 2: Available Stage Comparison [Legacy ADLS Gen 2]:Additionally, the ADLS Gen 2 Legacy stage which used Hadoop FS is also still available to be utilized.** It should be noted though that when exporting a pipeline made on a version prior to 5.5 into later Data Collectors that the (Legacy) tag will not be present on the stage information. However, the description of the stage will still relay the method with which the resource is accessed. When the (Legacy) stage is selected in a current pipeline it can be identified as seen in the image below

19 days ago

blakeorsakFan

posted in Community Articles and Got a Question?

Excessively large engine-$$.out file in the /tmp directory

The Issue: Data Collector has created a file in the /tmp directory that is growing to an excessive size and is taking up space. The purpose of this file is to serve as a location for stdout from a running Data Collector. Under normal circumstances, this file does not receive much data and should not grow to a large size. Specifically, the rotatelogs is used to limit the size of this file to ~10mb. However, there are a few instances where this file will receive more data than usual, or grow to an excessively large size. Adding certain Java parameters to your $SDC_HOME/libexec/sdc-env.sh file can result in extra output being sent to this /tmp file. This output is often useful for debugging, but can fill the file up quickly. An example of this might be the -Djavax.net.debug=ssl,handshake parameter used to debug SSL connections. If the rotatelogs utility is not currently available on your machine, the /tmp/engine-$$.out file may continue to grow past the limited size of ~10mb. The So

19 days ago

SindhuStreamSets Employee

posted in Share your Best Practices

Pipeline ID change.

Does the pipeline ID change, when any changes are made to the pipeline and re-published ?Pipeline ID is assigned during pipeline creation and does not change under any circumstances.

19 days ago

FranciscoFan

asked in Community Articles and Got a Question?

HTTP Processor for POST request clearing header values

Hello! I’m using record header attributes as variables. For example, during pipeline execution, saving ${time:now()} in a header attribute with an Expression Evaluator, to be used in other stages. This is no problem for HTTP GET requests, as the data gets aggregated into existing records. The problem is with HTTP Processor for POST requests, where the response overwrites all existing data from before, body and headers.Is there anyway around this? I’m trying to use this header attribute value that I saved to be used in anohter HTTP POST request after the one I just mentioned, but the value is lost. Thanks in advance!

20 days ago

AkshayJadhavStreamSets Employee

posted in Share your Best Practices

Configuring Google Secret Manager Credential Store for data collector.

In this article, we will walk you through the necessary steps to configure and set up Google Secrets for use with Data Collector.Firstly, We have to create the secret on the GCP which we will be using in the pipeline. In this article, I am going to store the mysql username and password in secrets.These are the secret created in GCP:- The pre-requisite is to install the package Google Secret Manager Credential Store on the Data collector and then update the following properties in credential-stores.properties file.credentialStores=gcp credentialStore.gcp.config.project.id= <gcp-projectID> credentialStore.gcp.config.credentialsMode=jsonPath credentialStore.gcp.config.credentialsJsonPath=/Users/apple/.config/gcloud/application_default_credentials.jsonOnce done, restart the data collector. This implicit EL expression we are going to use in the pipeline credentials pipeline:Username => ${credential:get("gcp", "all@all", "akshay_sqlusername?latest")}Password => ${credential:get("

26 days ago

paulStreamSets Employee

posted in Community Articles and Got a Question?

SDC 5.7.x - Orchestration pipeline calling a Job Template does not function properly

Environment:StreamSets Data Collector (SDC) 5.7.0 & 5.7.1 An orchestration pipeline which uses Start Jobs stages The orchestration stage starts a job using a Job Template The orchestration stage is configured with the Run in Background setting disabled Issue:In StreamSets Data Collector (SDC) 5.7.0 and 5.7.1, orchestration pipelines that use a Start Jobs stage to start a job using a Job Template do not function properly when Run in Background is disabled. With this configuration, the pipeline should wait for the job to complete before moving to the next stage, but instead the pipeline starts the job and immediately moves on, which can cause unexpected results for downstream jobs. Resolution:There is a fix planned for this issue in the SDC 5.8.0 release. In the short term, if your organization has an active support contact, you can contact StreamSets Support to obtain a patch to resolve the issue.

26 days ago

AkshayJadhavStreamSets Employee

posted in Share your Best Practices

SDC 5.7.X is not honouring the proxy configurations.

Environment:Streamsets Data Collector 5.7.X Proxy Configurations. Issue:There seems to be an issue with the latest version of the data collector (5.7.x). Specifically, it has come to our attention that the "http.nonProxyHosts" configuration is not being honored. Additionally, when executing the "ps -ef | grep sdc" command on the Linux host, the Java configurations are not visible. Solution:There is an issue with the data collector where proxy configurations are not working and will be fixed in further version(will keep updating the article once we have the exact fix version).The alternative/workaround is to just set it in the “Advanced Java Options” panel:-Dhttp.nonProxyHosts='*.azuresynapse.net'Any JVM configs specified directly will override those configurations and you could confirm by running “ps -ef | grep sdc” command on the Linux machine where data collector is running. NOTE: We have made some changes in latest version of data collector not to expose the proxy configuration hen

27 days ago

Page 1 / 44

Badge winners

Sanjeevhas earned the badge Eager to help
AkshayJadhavhas earned the badge Eager to help
john.durkinhas earned the badge Eager to help
samhas earned the badge Eager to help
saleempothiwalahas earned the badge Eager to help

Show all badges

Terms & Conditions

Sign up

Already have an account? Login

Social Login

Username *

E-mail address *

What I do... *

Data Leader Data Architect Data Engineer Data Scientist Other

Company *

Country *

Zip Code *

Marketing Communications

Yes No

Password *

I have read and Agree to the Website Terms of Service and I have read and acknowledged the Privacy Policy.

loginBox.register.email_repeat

Login to the community

No account yet? Create an account

Social Login

Username or Email

Password

Remember me

Forgot password?

Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.

Username or e-mail

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.