Skip to main content

StreamSets Platform

Get inspired and gain all the knowledge you need

1217 Topics
D
Fan
dhanraj_shindeFan
asked in Community Articles and Got a Question?
Tarball engine installation

Hi Team While installing DC Tarball engine using install script from Dataops platform, it asks for download and install directory at run time.Is there a way we could avoid passing it during execution and pass it in install script or use current directory? 

211
XavierV
StreamSets Employee
10 months ago
Clément Vi
Fan
Clément ViFan
asked in Community Articles and Got a Question?
Kafka consumer Offset from beginning every batch

Hello,I'm currently working on a simple pipeline to ingest kafka messages inside a log file.I'm trying to consume all the data from the beginning of a topic but i'm only getting newer data added to this topic.Once consumed, the previous topic messages are not accessible anymore.I've already test all the different "Auto Offset Reset" properties.Same for simple and multi topic consumers.In the official documentation docs.streamsets.com :auto.commit.interval.ms bootstrap.servers enable.auto.commit group.id max.poll.recordsIf I understand correctly all those parameters are locked so I can't disable the offset management and process all the data from the beginning of a topic.Is there an additionnal Kafka configuration property to use or do I need to configure the topic directly via kafka CLI ??StreamSets Data Collector version : 3.14.0Kafka Consumer version : 2.0.0Regards.

1624
Clément Vi
Fan
10 months ago
D
Discovered Fame
DolphinDiscovered Fame
asked in StreamSets Academy
How to set and get pipeline runtime configuration parameter value

Hi Team,We have some pipelines, which has some configuration like below, How can I get the exact plain test value for these parameter property value? Also how and where can I set these kind of values?   

503
Bikram
Headliner
10 months ago
A
Fan
andalvFan
asked in Community Articles and Got a Question?
How to retain original file name when writing to s3?

I read from an SFTP server and when writing to S3 my files get renamed. How can I retain the original file name?

1022
Bikram
Headliner
10 months ago
Y
Discovered Fame
yogesh0590
asked in Community Articles and Got a Question?
Update “globalMaxRetries” job config using REST API Call

Hello Team,I need to update one job config which is “globalMaxRetries” using rest call.I refer Restful section in Control Hub but that Rest call for “updateJob” requires many other configs also needs to mention in Request Body. could you please assist me with this ?

924
shankaravi
Fan
10 months ago
D
Roadie
drozdse1Roadie
asked in Share your Best Practices
Microsoft CDC Client

Hi,We are using Microsoft SQL CDC Client to extract changes on one of our database. Now we noticed that the target (a Snowflake destination) is complaining about unknown operations.So added this following step to the data flow:       if (records[i].attributes['sdc.operation.type'] == 5)         continue;         sdc.output.write(records[i]);Is it normal that we have to filter out operation type 5 manually? So that the pipeline works.Is there an example pipeline (SQL CDC → Snowflake) so that we could apply best practicesThanksSebastian

110
D
Roadie
10 months ago
S
Roadie
shivrajRoadie
asked in Community Articles and Got a Question?
Release connection Port of vm instace, used by pipeline in which Http server origin is used

I have developed a pipeline that utilizes an Http Server as the origin. In the configuration of the origin, I have designated port 8000 for the purpose of writing data to the pipeline. However, the issue lies in the fact that the pipeline does not release this port. I aim to utilize the same port for multiple pipelines, distinguishing them by their respective Application-Id. Upon executing the initial pipeline, the port remains occupied and fails to release. Upon attempting to execute a second pipeline using the same origin, I encountered an error message stating "Port is not free."If anyone got solution, Please Help me out with this.

282
S
Roadie
10 months ago
Hamid K
Fan
Hamid KFan
asked in Community Articles and Got a Question?
JSON payload validation using json schema

Hello community, I am vuilding a pipeline that receives http requests providing a JSON payload in the body.I want to validate the payload against a json schema. Now I have come accross this article which is interesting https://streamsets.com/blog/json-validator/. I keep getting this error : JSON_VAL_02 - The JSON object supplied is invalid: org.json.JSONException: Expected a ':' after a key at 6 [character 7 line 1] When I check the record as json string checkbox, it raises the exception … com.streamsets.pipeline.api.impl.TypeSupportConversionException: Cannot convert Map to com.streamsets.pipeline.api.impl.StringTypeSupport@374848aAlso with that processor component I am restricted as I would like to fetch the json schema from a repository.I have tried using the javascript evaluator as well and also there I am running into trouble with library versions I believe …Below the error I am getting …Caused by: java.io.IOException: resource /draftv4/schema not foundUsing libraries …json-schem

1470
Hamid K
Fan
10 months ago
Hamid K
Fan
Hamid KFan
asked in Community Articles and Got a Question?
Testing HTTP server origin on SDC deployed in docker

Hello all,I am trying to run a pipeline with an HTTP server origin to receive messages and then push them to a kafka topic. When I start my docker container, the port is not accessible on my local laptop when I don’t enable port-forwarding for the port the origin is listening on. But when I enable port-forwarding then I get the notification : “The engine cannot be reached. Check that the engine is running and can communicate with Control Hub.” Can anyone help me out on this?Many thanks!Hamid K.

351
Hamid K
Fan
10 months ago
F
Fan
fhermosillaFan
asked in Community Articles and Got a Question?
Using GCP EL credentials in a custom function

Hi Everyone.We are working in a solution which integrates GCP Secret Manager and Streamsets.The client stores a secret in JSON format, and wants to extract a field with his data. Is similar to this format:   How can I extract the dbuser field??I now there are custom functions, but don’t know if I can use the EL function from Streamsets to obtain the JSON info.Can I use the ${credential:get("gcp", "group@org", "dbuser?latest") as an argument? so the function returns only the dbuser value. Thanks! 

140
F
Fan
10 months ago
hrishikeh132609
Fan
hrishikeh132609Fan
asked in Community Articles and Got a Question?
Teradata to Snowflake (Extract and Load)

Hi,Use case.We are load data from Teradata to Snowflake.as we not doing any transformation just extract the Data and loading in Snowflake.which engine will be the best for this use case.1.Data collector​​​​​​​2.Transformer for spark.Note: we have data in millions  Best regards,Hrishikesh 

150
hrishikeh132609
Fan
10 months ago
N
Fan
nachiket_petheFan
asked in Community Articles and Got a Question?
After enabling https on Control Hub, login screen redirects back to login screen.

We have performed following steps according to this documentation - https://docs.streamsets.com/portal/controlhub/latest/onpremhelp/controlhub/UserGuide/Install/EnableHTTPS.html1. stored .p12 file in etc/dpm directory2. made changes in dpm.properties (exactly as given in the documentation)3. made changes in common-to-all.properties (exactly as given in the documentation)4. stored keystore password in keystore-password.txt5. start the control hub manuallywe are using control hub version 3.51.4

200
N
Fan
11 months ago
pedro
StreamSets Employee
pedroStreamSets Employee
posted in Community Articles and Got a Question?
How to use Content Type ‘multipart/form-data’ in the HTTP Client stage

SDC’s HTTP Client stage does not have a native option to send HTTP requests using the Content Type ‘multipart/form-data’. Using the default options will result in SDC making the call seemingly correctly but it will not add the required boundary characters that a request using this kind of Content Type requires. However, we can force SDC to send the call in the correct multipart/form-data’ format. To do this you will need to set these parameters in the following way:*** For this example, the request SDC would make is equivalent to the following curl command:curl --location 'https://test.url.com/api/jobs' \--header 'Accept: application/json' \--header 'Authorization: Bearer *********************' \--header 'Cookie: staging-connections-canary="45412ba154f512d34"' \--form 'options="{\"options\":{},\"url\":\"https://test.url.com\",\"sslVersion\":\"TLS 1.2\",\"password\":xxxxxxxxxx,\"userId\":xxxxxxxxxx,\"displayName\":\"Pedro\",\"jobName\":\"pedro_test_job\",\"batchsize\":1000,\"waitTimeout

13772
N
Opening Band
11 months ago
N
Opening Band
N_aliOpening Band
asked in Community Articles and Got a Question?
Move Files from sftp server to httpclient dst as an attachment?

Hi , i need to transfer csv files from an sftp server to post it as an attachment in a post API (http client),Can you help me figure out how to accomplish this ?

70
N
Opening Band
11 months ago
Pradeep
StreamSets Employee
PradeepStreamSets Employee
asked in Community Articles and Got a Question?
API which returns all pipelines

Problem description: By default https://<control-hub or platform endpoint>:<port-number>/pipelinestore/rest/v1/pipelines returns the pipelines which are committed. Pipeline count which you note in below image is the number committed pipelines. What if we want the pipelines api to return non-committed or draft pipelines too?

201
Pradeep
StreamSets Employee
11 months ago
john.mcavoy
StreamSets Employee
john.mcavoyStreamSets Employee
posted in Community Articles and Got a Question?
UTF-8 Special Characters being replaced by Question Marks (�)in Pipelines

Problem DescriptionWhen running an SDC pipeline which processes records with Strings containing UTF-8 special characters, these special characters are being replaced by question marks (? or �) in various parts of your pipeline.ExampleInput record:{"productName": "My Product™"}Output record:{"productName": "My Product�"}Root CauseThis problem indicates that your Java Runtime Environment is using a non-UTF-8 character set which does not support these special characters, so the special characters are being replaced by the � (U+FFFD) REPLACEMENT CHARACTER.By default, SDC will try to set your JAVA_OPTS to use the UTF-8 encoding, however it is possible that the JAVA_OPTS parameters are being set outside of SDC and are overriding either of the following JVM parameters which tells the JRE what encoding to use at runtime: file.encoding and sun.jnu.encoding.The JRE also has a default character set which it will use if these two parameters are not specified. This default character set can vary be

87581
D
Roadie
11 months ago
Douglas R
Fan
Douglas RFan
posted in Community Articles and Got a Question?
"Failed at Step EXEC - Permission denied" when starting SDC as a Service on Systems with SELinux

"Failed at step EXEC  - Permission denied" when starting SDC as a Service on Systems with SELinux ProblemWhen starting StreamSets Data Collector as a service under systemd, the service fails immediately on startup.  The following error is shown when the status of the service is checked (using the systemctl status sdc command).systemd[13801]: sdc.service: Failed at step EXEC spawning /opt/streamsets-datacollector/bin/streamsets: Permission denied CauseThis error indicates that the script SDC_HOME/bin/streamsets (which is used to start SDC) could not be launched by systemd.   This can be caused by incorrect file or directory permissions or by problems with the SELinux context (on systems where SELinux is enabled). SolutionStep 1:  Examine the Service ConfigurationBefore looking at possible permissions issues, it is important to verify that the user, group, and installation directory for the service are correctly configured in the service unit file, /etc/systemd/system/sdc.service.  Exami

89351
devops18
Fan
11 months ago
mfalcon
mfalconNew Member
asked in Community Articles and Got a Question?
Export Job error

When I’m trying to export a job it shows the following response{"ISSUES":[{"code":"CONNECTION_IMPORT_EXPORT_02","message":"Failed to export connections metadata, error is Issues: [CONNECTION_IMPORT_EXPORT_01 - Cannot retrieve the results from the connections application (HTTP Code 400)]"}]}'It’s only hapenning with jobs that doens’t have any data on the Parameters tab of Configuration section.Does anybody now why is happening this?

410
mfalcon
11 months ago
san
Fan
sanFan
asked in Community Articles and Got a Question?
StreamSet with MS SQL connection Issues

Hi The Top Contributors, I have tried many connections on SQL with StreamSet but it is not workingsRemediation steps:Make sure the connection properties have been properly set to allow SSL connections with the database. Check out the links below for more information on how to configure the SQL Server properties. - https://learn.microsoft.com/en-us/sql/connect/jdbc/connecting-with-ssl-encryption?view=sql-server-ver15#configuring-the-connection - https://learn.microsoft.com/en-us/sql/connect/jdbc/setting-the-connection-properties?view=sql-server-ver15 Technical details:The SSL negotiation has failed and the driver found has not been able to initialize the Connection Pool: Failed to initialize pool: "encrypt" property is set to "true" and "trustServerCertificate" property is set to "false" but the driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption: Error: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderEx

2012
antmcmullen
StreamSets Employee
11 months ago
I
Fan
ingridmirnikFan
asked in StreamSets Academy
Why does self-managed server keep crashing on engines restart and what are proper transfomer configurations?

We‘ve set up a machine for Self-Managed Deployments (specifications below) with Docker as it’s said in the documentation. Machine specifications:Operating system: Linux (ubuntu 22.04) Disk size: 30GiB RAM: 4GiB Performance tier: P10 - 500 IOPS, 100MMB/sWe succesfully created some Data Collector and Transfomer engines. The problem occurs when we try to restart/run all of the engines at the same time. Server just freezes and eventually stops responding.If you look at the metrics of the machine, you can see that it stops usually when “Disk Bytes” peek. My first question is, what could cause this huge usage of resources to crash the server eventually? The next things are Transformer engines. In documentation it says:“Important: When using a Transformer engine that works with a Spark cluster, edit the transformer.base.http.url property on the Transformer Configuration tab. Uncomment the property and set it to the Transformer URL. For more information, see Granting the Spark Cluster Access t

250
I
Fan
1 year ago
aark
aarkNew Member
asked in Community Articles and Got a Question?
Pagination to a streamsets pipeline throws null pointer exception

Trying to add pagination to a streamsets pipeline that requests data from Tableau. Currently, the API is of the format: ttps://us-west-2b.online.tableau.com/api/3.3/sites/${record:value('/credentials/site/id')}/workbooks?pageNumber=${startAt} , where startAt starts at 1 and should go to 4. I selected the "By Page Number" pagination mode option with an offset of 1. However, I am getting a null pointer exception on making the request. 

130
aark
1 year ago
J
Opening Band
jyuanOpening Band
asked in Community Articles and Got a Question?
Can the temporary directories be places in another location

We are running the SDC on Linux servers. I noticed that many temporary directories are created under the /tmp directory. I assume that this is the default location for these sub-directories. However, I would like to use a different location for them. For example, maybe I can use /data/tmp?What’s the procedure to specify another location for those temporary sub-directories?Thanks!

564
J
Opening Band
1 year ago
tejas_surve
Roadie
tejas_surveRoadie
asked in Community Articles and Got a Question?
How to use Azure Service Bus ?

How to use Azure service Bus as messaging queue in StreamSets?How to configure Connection?

330
tejas_surve
Roadie
1 year ago
S
Roadie
shivrajRoadie
asked in Community Articles and Got a Question?
How to connect Hazelcast MQ with Streamsets Pipeline?

I want to establish a connect between Hazelcast-MQ to use it as origin and destination. Couldn’t find any information about that in Streamsets Documentation. Please help me out with this.

381
Dimas Cabré i Chacón
StreamSets Employee
1 year ago
Francisco
Fan
FranciscoFan
asked in Community Articles and Got a Question?
JDBC Query Consumer SSL error

Hello all, I had a pipeline that was running well with a JDBC Query Consumer stage with a simple select to a MSSQL database server, until one day I started getting the following error:DBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: "Certificates do not conform to algorithm constraints". ClientConnectionId:1115d9c7-5599-411e-ba8e-3c379d2ac0ac: hikariConfigBean.connectionString Is this related to changes that may have been made in the database server? There is no TLS configuration on this stage, I’m not sure if it’s using TLS by default. Checking with the database owner, he indicated that the SQL Server is compatible with TLS 1.2. Any help is appreciated, thank you.

670
Francisco
Fan
1 year ago
Terms & Conditions