Skip to main content

- - Knowledge base
Product Updates
Events

30-Day Free Trial: It’s Never Been Easier To Get Started With StreamSets

A

1 year ago

Home
Community overview
StreamSets Platform
Community Articles and Got a Question?

Community Articles and Got a Question?

Can't find what you're looking for? Ask it here or check out the Community articles

1,033 Topics
2,323 Replies

When you subscribe we will email you when there is a new topic in this category

1033 Topics

Recently active

Most replies Most views Newest first

dhanraj_shindeFan

asked in Community Articles and Got a Question?

Tarball engine installation

Hi Team While installing DC Tarball engine using install script from Dataops platform, it asks for download and install directory at run time.Is there a way we could avoid passing it during execution and pass it in install script or use current directory?

asked in Community Articles and Got a Question?

Kafka consumer Offset from beginning every batch

Hello,I'm currently working on a simple pipeline to ingest kafka messages inside a log file.I'm trying to consume all the data from the beginning of a topic but i'm only getting newer data added to this topic.Once consumed, the previous topic messages are not accessible anymore.I've already test all the different "Auto Offset Reset" properties.Same for simple and multi topic consumers.In the official documentation docs.streamsets.com :auto.commit.interval.ms bootstrap.servers enable.auto.commit group.id max.poll.recordsIf I understand correctly all those parameters are locked so I can't disable the offset management and process all the data from the beginning of a topic.Is there an additionnal Kafka configuration property to use or do I need to configure the topic directly via kafka CLI ??StreamSets Data Collector version : 3.14.0Kafka Consumer version : 2.0.0Regards.

asked in Community Articles and Got a Question?

How to retain original file name when writing to s3?

I read from an SFTP server and when writing to S3 my files get renamed. How can I retain the original file name?

asked in Community Articles and Got a Question?

Update “globalMaxRetries” job config using REST API Call

Hello Team,I need to update one job config which is “globalMaxRetries” using rest call.I refer Restful section in Control Hub but that Rest call for “updateJob” requires many other configs also needs to mention in Request Body. could you please assist me with this ?

asked in Community Articles and Got a Question?

Release connection Port of vm instace, used by pipeline in which Http server origin is used

I have developed a pipeline that utilizes an Http Server as the origin. In the configuration of the origin, I have designated port 8000 for the purpose of writing data to the pipeline. However, the issue lies in the fact that the pipeline does not release this port. I aim to utilize the same port for multiple pipelines, distinguishing them by their respective Application-Id. Upon executing the initial pipeline, the port remains occupied and fails to release. Upon attempting to execute a second pipeline using the same origin, I encountered an error message stating "Port is not free."If anyone got solution, Please Help me out with this.

asked in Community Articles and Got a Question?

JSON payload validation using json schema

Hello community, I am vuilding a pipeline that receives http requests providing a JSON payload in the body.I want to validate the payload against a json schema. Now I have come accross this article which is interesting https://streamsets.com/blog/json-validator/. I keep getting this error : JSON_VAL_02 - The JSON object supplied is invalid: org.json.JSONException: Expected a ':' after a key at 6 [character 7 line 1] When I check the record as json string checkbox, it raises the exception … com.streamsets.pipeline.api.impl.TypeSupportConversionException: Cannot convert Map to com.streamsets.pipeline.api.impl.StringTypeSupport@374848aAlso with that processor component I am restricted as I would like to fetch the json schema from a repository.I have tried using the javascript evaluator as well and also there I am running into trouble with library versions I believe …Below the error I am getting …Caused by: java.io.IOException: resource /draftv4/schema not foundUsing libraries …json-schem

asked in Community Articles and Got a Question?

Testing HTTP server origin on SDC deployed in docker

Hello all,I am trying to run a pipeline with an HTTP server origin to receive messages and then push them to a kafka topic. When I start my docker container, the port is not accessible on my local laptop when I don’t enable port-forwarding for the port the origin is listening on. But when I enable port-forwarding then I get the notification : “The engine cannot be reached. Check that the engine is running and can communicate with Control Hub.” Can anyone help me out on this?Many thanks!Hamid K.

asked in Community Articles and Got a Question?

Using GCP EL credentials in a custom function

Hi Everyone.We are working in a solution which integrates GCP Secret Manager and Streamsets.The client stores a secret in JSON format, and wants to extract a field with his data. Is similar to this format: How can I extract the dbuser field??I now there are custom functions, but don’t know if I can use the EL function from Streamsets to obtain the JSON info.Can I use the ${credential:get("gcp", "group@org", "dbuser?latest") as an argument? so the function returns only the dbuser value. Thanks!

hrishikeh132609Fan

asked in Community Articles and Got a Question?

Teradata to Snowflake (Extract and Load)

Hi,Use case.We are load data from Teradata to Snowflake.as we not doing any transformation just extract the Data and loading in Snowflake.which engine will be the best for this use case.1.Data collector2.Transformer for spark.Note: we have data in millions Best regards,Hrishikesh

nachiket_petheFan

asked in Community Articles and Got a Question?

After enabling https on Control Hub, login screen redirects back to login screen.

We have performed following steps according to this documentation - https://docs.streamsets.com/portal/controlhub/latest/onpremhelp/controlhub/UserGuide/Install/EnableHTTPS.html1. stored .p12 file in etc/dpm directory2. made changes in dpm.properties (exactly as given in the documentation)3. made changes in common-to-all.properties (exactly as given in the documentation)4. stored keystore password in keystore-password.txt5. start the control hub manuallywe are using control hub version 3.51.4

pedroStreamSets Employee

posted in Community Articles and Got a Question?

How to use Content Type ‘multipart/form-data’ in the HTTP Client stage

SDC’s HTTP Client stage does not have a native option to send HTTP requests using the Content Type ‘multipart/form-data’. Using the default options will result in SDC making the call seemingly correctly but it will not add the required boundary characters that a request using this kind of Content Type requires. However, we can force SDC to send the call in the correct multipart/form-data’ format. To do this you will need to set these parameters in the following way:*** For this example, the request SDC would make is equivalent to the following curl command:curl --location 'https://test.url.com/api/jobs' \--header 'Accept: application/json' \--header 'Authorization: Bearer *********************' \--header 'Cookie: staging-connections-canary="45412ba154f512d34"' \--form 'options="{\"options\":{},\"url\":\"https://test.url.com\",\"sslVersion\":\"TLS 1.2\",\"password\":xxxxxxxxxx,\"userId\":xxxxxxxxxx,\"displayName\":\"Pedro\",\"jobName\":\"pedro_test_job\",\"batchsize\":1000,\"waitTimeout

N_aliOpening Band

asked in Community Articles and Got a Question?

Move Files from sftp server to httpclient dst as an attachment?

Hi , i need to transfer csv files from an sftp server to post it as an attachment in a post API (http client),Can you help me figure out how to accomplish this ?

PradeepStreamSets Employee

asked in Community Articles and Got a Question?

API which returns all pipelines

Problem description: By default https://<control-hub or platform endpoint>:<port-number>/pipelinestore/rest/v1/pipelines returns the pipelines which are committed. Pipeline count which you note in below image is the number committed pipelines. What if we want the pipelines api to return non-committed or draft pipelines too?

john.mcavoyStreamSets Employee

posted in Community Articles and Got a Question?

UTF-8 Special Characters being replaced by Question Marks (�)in Pipelines

Problem DescriptionWhen running an SDC pipeline which processes records with Strings containing UTF-8 special characters, these special characters are being replaced by question marks (? or �) in various parts of your pipeline.ExampleInput record:{"productName": "My Product™"}Output record:{"productName": "My Product�"}Root CauseThis problem indicates that your Java Runtime Environment is using a non-UTF-8 character set which does not support these special characters, so the special characters are being replaced by the � (U+FFFD) REPLACEMENT CHARACTER.By default, SDC will try to set your JAVA_OPTS to use the UTF-8 encoding, however it is possible that the JAVA_OPTS parameters are being set outside of SDC and are overriding either of the following JVM parameters which tells the JRE what encoding to use at runtime: file.encoding and sun.jnu.encoding.The JRE also has a default character set which it will use if these two parameters are not specified. This default character set can vary be

posted in Community Articles and Got a Question?

"Failed at Step EXEC - Permission denied" when starting SDC as a Service on Systems with SELinux

"Failed at step EXEC - Permission denied" when starting SDC as a Service on Systems with SELinux ProblemWhen starting StreamSets Data Collector as a service under systemd, the service fails immediately on startup. The following error is shown when the status of the service is checked (using the systemctl status sdc command).systemd[13801]: sdc.service: Failed at step EXEC spawning /opt/streamsets-datacollector/bin/streamsets: Permission denied CauseThis error indicates that the script SDC_HOME/bin/streamsets (which is used to start SDC) could not be launched by systemd. This can be caused by incorrect file or directory permissions or by problems with the SELinux context (on systems where SELinux is enabled). SolutionStep 1: Examine the Service ConfigurationBefore looking at possible permissions issues, it is important to verify that the user, group, and installation directory for the service are correctly configured in the service unit file, /etc/systemd/system/sdc.service. Exami

mfalconNew Member

asked in Community Articles and Got a Question?

Export Job error

When I’m trying to export a job it shows the following response{"ISSUES":[{"code":"CONNECTION_IMPORT_EXPORT_02","message":"Failed to export connections metadata, error is Issues: [CONNECTION_IMPORT_EXPORT_01 - Cannot retrieve the results from the connections application (HTTP Code 400)]"}]}'It’s only hapenning with jobs that doens’t have any data on the Parameters tab of Configuration section.Does anybody now why is happening this?

sanFan

asked in Community Articles and Got a Question?

StreamSet with MS SQL connection Issues

Hi The Top Contributors, I have tried many connections on SQL with StreamSet but it is not workingsRemediation steps:Make sure the connection properties have been properly set to allow SSL connections with the database. Check out the links below for more information on how to configure the SQL Server properties. - https://learn.microsoft.com/en-us/sql/connect/jdbc/connecting-with-ssl-encryption?view=sql-server-ver15#configuring-the-connection - https://learn.microsoft.com/en-us/sql/connect/jdbc/setting-the-connection-properties?view=sql-server-ver15 Technical details:The SSL negotiation has failed and the driver found has not been able to initialize the Connection Pool: Failed to initialize pool: "encrypt" property is set to "true" and "trustServerCertificate" property is set to "false" but the driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption: Error: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderEx

aarkNew Member

asked in Community Articles and Got a Question?

Pagination to a streamsets pipeline throws null pointer exception

Trying to add pagination to a streamsets pipeline that requests data from Tableau. Currently, the API is of the format: ttps://us-west-2b.online.tableau.com/api/3.3/sites/${record:value('/credentials/site/id')}/workbooks?pageNumber=${startAt} , where startAt starts at 1 and should go to 4. I selected the "By Page Number" pagination mode option with an offset of 1. However, I am getting a null pointer exception on making the request.

jyuanOpening Band

asked in Community Articles and Got a Question?

Can the temporary directories be places in another location

We are running the SDC on Linux servers. I noticed that many temporary directories are created under the /tmp directory. I assume that this is the default location for these sub-directories. However, I would like to use a different location for them. For example, maybe I can use /data/tmp?What’s the procedure to specify another location for those temporary sub-directories?Thanks!

tejas_surveRoadie

asked in Community Articles and Got a Question?

How to use Azure Service Bus ?

How to use Azure service Bus as messaging queue in StreamSets?How to configure Connection?

asked in Community Articles and Got a Question?

How to connect Hazelcast MQ with Streamsets Pipeline?

I want to establish a connect between Hazelcast-MQ to use it as origin and destination. Couldn’t find any information about that in Streamsets Documentation. Please help me out with this.

asked in Community Articles and Got a Question?

JDBC Query Consumer SSL error

Hello all, I had a pipeline that was running well with a JDBC Query Consumer stage with a simple select to a MSSQL database server, until one day I started getting the following error:DBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: "Certificates do not conform to algorithm constraints". ClientConnectionId:1115d9c7-5599-411e-ba8e-3c379d2ac0ac: hikariConfigBean.connectionString Is this related to changes that may have been made in the database server? There is no TLS configuration on this stage, I’m not sure if it’s using TLS by default. Checking with the database owner, he indicated that the SQL Server is compatible with TLS 1.2. Any help is appreciated, thank you.

bonthunagireddy03Fan

asked in Community Articles and Got a Question?

How to connect to IBM MQ using SDC JMS origin

Hi Team,we are trying to establish a connection to IBM MQ using JMS origin, but we couldn’t be able to do as could not find much information on StreamSets documentation (https://docs.streamsets.com/portal/#datacollector/latest/help/datacollector/UserGuide/Origins/JMS.html?hl=jms). Do you have any reference document which says how to establish a connection to IBM MQ? much thanks in advance!!

john.mcavoyStreamSets Employee

asked in Community Articles and Got a Question?

How to Cleanup Job History for SDC and SCH

ProblemOld SCH job run histories are taking up a lot of disk space either in SDC nodes or in SCH’s DB, and these histories need to be limited for better performance. SolutionJob run history is stored in SCH and in SDC (if SDC has configured dpm.runHistory.enabled=true in $SDC_CONF/dpm.properties).In SCH, you can configure Maximum number of job runs for each Organization. This is the max number of runs retained for each job's history SCH periodically runs a clean-up task to purge all old job histories when this limit is reached. SDC can also retain a local copy of job run histories if dpm.runHistory.enabled=true is configured in $SDC_CONF/dpm.properties. SDC keeps runtime information pertaining to each running pipeline in $SDC_DATA/runInfo. SDC deletes any local job history information in $SDC_DATA/runInfo whenever a pipeline finishes. If you set dpm.runHistory.enabled=true, then SDC will copy each job’s history from $SDC_DATA/runInfo into $SDC_DATA/runHistory before this clean-up oc

somFan

posted in Community Articles and Got a Question?

Socket Timeout Exceptions when using "Start Job" stage.

Environment:* StreamSets Control Hub 3.19.2* SDC server 5.2.0* Transformer 4.3.0-0049ISSUE:Start Jobs stage failing in 5.2.0 with: java.net.SocketTimeoutException: Read timed outSOLUTION:Data Collector v5.2.0 sets a default read timeout of 10 seconds.Control Hub logs show numerous jobs take more than 10 seconds to start, with the highest being 42 seconds.Customer to update read timeout setting to a higher value, such as 60 or 90 seconds to fix this issue in this scenario.Click on the Start Jobs stage Click on the Configuration in the left menu Click on the HTTP tab Change Read Timeout (ms) from 10000 to another value, such as 60000 or 90000

1
2
3
4
5
6
...
42

Page 3 / 42

Badge winners

Ykyeeywkkwhas earned the badge Eager to help
vishwesh.margasahayamhas earned the badge Product expert
ajinkyahas earned the badge Innovator
Sanjeevhas earned the badge Eager to help
AkshayJadhavhas earned the badge Eager to help

Show all badges

Powered by Gainsight

Terms & Conditions

Sign up

Already have an account? Login

Social Login

or

Username *

E-mail address *

What I do... *

Data Leader Data Architect Data Engineer Data Scientist Other

Company *

Country *

Zip Code *

Marketing Communications

Yes No

Password *

I have read and Agree to the Website Terms of Service and I have read and acknowledged the Privacy Policy.

loginBox.register.email_repeat

Login to the community

No account yet? Create an account

Social Login

or

Username or Email

Password

Remember me

Forgot password?

Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.

Enter your e-mail address

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

OK

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.

OK