Skip to main content

Community Articles and Got a Question?

Can't find what you're looking for? Ask it here or check out the Community articles

  • 1,040 Topics
  • 2,326 Replies
1040 Topics
J
Fan
jangala.karthikFan
asked in Community Articles and Got a Question?
Change Runtime Parameter value

I have passed a parameter to my pipeline which has a default value when I start the pipeline. I need to change it's value in from one of processors. Is it possible to change the runtime parameter? If yes, can you please provide the steps to do that.

10636
K
Fan
8 months ago
L
Fan
LeoG7119Fan
asked in Community Articles and Got a Question?
SNOWFLAKE_74 - Using EL requires auto-creation error

I’m getting this error when I’m trying to use Snowflake uploader. I manually created the stage and I know I can put files there. I’m on 5.8 data collector.  How do I over come this error? 

333
Dimas Cabré i Chacón
StreamSets Employee
8 months ago
Skipper_609
Fan
Skipper_609Fan
asked in Community Articles and Got a Question?
Running Stored procedure and using the data returned by the SP

The case is simple,I have a Kafka topic to which the data gets pushed, I use the data to execute a stored procedure using the JDBC query node. This is working fine but I need to get the return value from the SP so I can log if the execution is a success or there was any error in input which is returned by the SP itself. The SP has some complex logic that involves multiple tables so it's not really good I think to eliminate the SP. Since I don't see any output in JDBC query executor how do I redo the pipeline so I can store/process the SP outputs?

350
Skipper_609
Fan
8 months ago
L
Fan
LeoG7119Fan
asked in Community Articles and Got a Question?
Snowflake runnable process batch failed: com.streamsets.pipeline.api.StageException: SNOWFLAKE_00 - Could not perform SQL operation: java.sql.SQLException: Error when adding primary keys:

I have a simple pipeline from oracle multi table into snowflake destination. This error only happens when the destination table is not present but it does create the table with PK. After pipeline ran and I can truncate the table and re-run the pipeline without any issues.  Any ideas? 

120
L
Fan
8 months ago
ino
Fan
inoFan
asked in Community Articles and Got a Question?
Java Throw Error when I using javascript evaluator in SDC

Hello!I'm currently trying out the free (open source version) of SDC. When I created an empty pipeline and added a cron scheduler and JavaScript Evaluator and tried to run it, I encountered the following error:Pipeline Status: STARTING_ERROR: java.lang.NoClassDefFoundError: Could not initialize class com.streamsets.pipeline.stage.processor.javascript.Java8JavaScriptObjectFactoryI was under the impression that using the JavaScript Evaluator did not require any additional package installations. Is there an additional step that needs to be taken?I need some help. Thank you.

441
Bikram
Headliner
8 months ago
R
Fan
ravvichaudharyFan
asked in Community Articles and Got a Question?
HTTP Client Processor Not Retrying Despite Configuration in StreamSets Data Collector

Hello,I'm currently using the HTTP Client Processor in StreamSets Data Collector and I've encountered an issue with the retry mechanism. Despite configuring the processor to retry on receiving certain HTTP status codes, it doesn't seem to be doing so.Specifically, I've set the processor to retry immediately when it receives a 409 status code, with a maximum of 2 retries. However, when the processor receives a 409 status code, it doesn't retry the request and instead gives the following error: "HTTP_101: Applying passthrough and error policy on status configuration".I've checked the pipeline configuration. I'm not sure why the processor isn't retrying as expected.Has anyone else encountered this issue? Any insights or suggestions would be greatly appreciated.Thank you in advance for your help.

360
R
Fan
9 months ago
D
Fan
dhanraj_shindeFan
asked in Community Articles and Got a Question?
Tarball engine installation

Hi Team While installing DC Tarball engine using install script from Dataops platform, it asks for download and install directory at run time.Is there a way we could avoid passing it during execution and pass it in install script or use current directory? 

201
XavierV
StreamSets Employee
9 months ago
Clément Vi
Fan
Clément ViFan
asked in Community Articles and Got a Question?
Kafka consumer Offset from beginning every batch

Hello,I'm currently working on a simple pipeline to ingest kafka messages inside a log file.I'm trying to consume all the data from the beginning of a topic but i'm only getting newer data added to this topic.Once consumed, the previous topic messages are not accessible anymore.I've already test all the different "Auto Offset Reset" properties.Same for simple and multi topic consumers.In the official documentation docs.streamsets.com :auto.commit.interval.ms bootstrap.servers enable.auto.commit group.id max.poll.recordsIf I understand correctly all those parameters are locked so I can't disable the offset management and process all the data from the beginning of a topic.Is there an additionnal Kafka configuration property to use or do I need to configure the topic directly via kafka CLI ??StreamSets Data Collector version : 3.14.0Kafka Consumer version : 2.0.0Regards.

1594
Clément Vi
Fan
9 months ago
A
Fan
andalvFan
asked in Community Articles and Got a Question?
How to retain original file name when writing to s3?

I read from an SFTP server and when writing to S3 my files get renamed. How can I retain the original file name?

952
Bikram
Headliner
9 months ago
Y
Discovered Fame
yogesh0590
asked in Community Articles and Got a Question?
Update “globalMaxRetries” job config using REST API Call

Hello Team,I need to update one job config which is “globalMaxRetries” using rest call.I refer Restful section in Control Hub but that Rest call for “updateJob” requires many other configs also needs to mention in Request Body. could you please assist me with this ?

914
shankaravi
Fan
9 months ago
S
Roadie
shivrajRoadie
asked in Community Articles and Got a Question?
Release connection Port of vm instace, used by pipeline in which Http server origin is used

I have developed a pipeline that utilizes an Http Server as the origin. In the configuration of the origin, I have designated port 8000 for the purpose of writing data to the pipeline. However, the issue lies in the fact that the pipeline does not release this port. I aim to utilize the same port for multiple pipelines, distinguishing them by their respective Application-Id. Upon executing the initial pipeline, the port remains occupied and fails to release. Upon attempting to execute a second pipeline using the same origin, I encountered an error message stating "Port is not free."If anyone got solution, Please Help me out with this.

282
S
Roadie
9 months ago
Hamid K
Fan
Hamid KFan
asked in Community Articles and Got a Question?
JSON payload validation using json schema

Hello community, I am vuilding a pipeline that receives http requests providing a JSON payload in the body.I want to validate the payload against a json schema. Now I have come accross this article which is interesting https://streamsets.com/blog/json-validator/. I keep getting this error : JSON_VAL_02 - The JSON object supplied is invalid: org.json.JSONException: Expected a ':' after a key at 6 [character 7 line 1] When I check the record as json string checkbox, it raises the exception … com.streamsets.pipeline.api.impl.TypeSupportConversionException: Cannot convert Map to com.streamsets.pipeline.api.impl.StringTypeSupport@374848aAlso with that processor component I am restricted as I would like to fetch the json schema from a repository.I have tried using the javascript evaluator as well and also there I am running into trouble with library versions I believe …Below the error I am getting …Caused by: java.io.IOException: resource /draftv4/schema not foundUsing libraries …json-schem

1470
Hamid K
Fan
9 months ago
Hamid K
Fan
Hamid KFan
asked in Community Articles and Got a Question?
Testing HTTP server origin on SDC deployed in docker

Hello all,I am trying to run a pipeline with an HTTP server origin to receive messages and then push them to a kafka topic. When I start my docker container, the port is not accessible on my local laptop when I don’t enable port-forwarding for the port the origin is listening on. But when I enable port-forwarding then I get the notification : “The engine cannot be reached. Check that the engine is running and can communicate with Control Hub.” Can anyone help me out on this?Many thanks!Hamid K.

351
Hamid K
Fan
9 months ago
F
Fan
fhermosillaFan
asked in Community Articles and Got a Question?
Using GCP EL credentials in a custom function

Hi Everyone.We are working in a solution which integrates GCP Secret Manager and Streamsets.The client stores a secret in JSON format, and wants to extract a field with his data. Is similar to this format:   How can I extract the dbuser field??I now there are custom functions, but don’t know if I can use the EL function from Streamsets to obtain the JSON info.Can I use the ${credential:get("gcp", "group@org", "dbuser?latest") as an argument? so the function returns only the dbuser value. Thanks! 

140
F
Fan
10 months ago
hrishikeh132609
Fan
hrishikeh132609Fan
asked in Community Articles and Got a Question?
Teradata to Snowflake (Extract and Load)

Hi,Use case.We are load data from Teradata to Snowflake.as we not doing any transformation just extract the Data and loading in Snowflake.which engine will be the best for this use case.1.Data collector​​​​​​​2.Transformer for spark.Note: we have data in millions  Best regards,Hrishikesh 

150
hrishikeh132609
Fan
10 months ago
N
Fan
nachiket_petheFan
asked in Community Articles and Got a Question?
After enabling https on Control Hub, login screen redirects back to login screen.

We have performed following steps according to this documentation - https://docs.streamsets.com/portal/controlhub/latest/onpremhelp/controlhub/UserGuide/Install/EnableHTTPS.html1. stored .p12 file in etc/dpm directory2. made changes in dpm.properties (exactly as given in the documentation)3. made changes in common-to-all.properties (exactly as given in the documentation)4. stored keystore password in keystore-password.txt5. start the control hub manuallywe are using control hub version 3.51.4

200
N
Fan
10 months ago
pedro
StreamSets Employee
pedroStreamSets Employee
posted in Community Articles and Got a Question?
How to use Content Type ‘multipart/form-data’ in the HTTP Client stage

SDC’s HTTP Client stage does not have a native option to send HTTP requests using the Content Type ‘multipart/form-data’. Using the default options will result in SDC making the call seemingly correctly but it will not add the required boundary characters that a request using this kind of Content Type requires. However, we can force SDC to send the call in the correct multipart/form-data’ format. To do this you will need to set these parameters in the following way:*** For this example, the request SDC would make is equivalent to the following curl command:curl --location 'https://test.url.com/api/jobs' \--header 'Accept: application/json' \--header 'Authorization: Bearer *********************' \--header 'Cookie: staging-connections-canary="45412ba154f512d34"' \--form 'options="{\"options\":{},\"url\":\"https://test.url.com\",\"sslVersion\":\"TLS 1.2\",\"password\":xxxxxxxxxx,\"userId\":xxxxxxxxxx,\"displayName\":\"Pedro\",\"jobName\":\"pedro_test_job\",\"batchsize\":1000,\"waitTimeout

13712
N
Opening Band
10 months ago
N
Opening Band
N_aliOpening Band
asked in Community Articles and Got a Question?
Move Files from sftp server to httpclient dst as an attachment?

Hi , i need to transfer csv files from an sftp server to post it as an attachment in a post API (http client),Can you help me figure out how to accomplish this ?

70
N
Opening Band
10 months ago
Pradeep
StreamSets Employee
PradeepStreamSets Employee
asked in Community Articles and Got a Question?
API which returns all pipelines

Problem description: By default https://<control-hub or platform endpoint>:<port-number>/pipelinestore/rest/v1/pipelines returns the pipelines which are committed. Pipeline count which you note in below image is the number committed pipelines. What if we want the pipelines api to return non-committed or draft pipelines too?

201
Pradeep
StreamSets Employee
10 months ago
john.mcavoy
StreamSets Employee
john.mcavoyStreamSets Employee
posted in Community Articles and Got a Question?
UTF-8 Special Characters being replaced by Question Marks (�)in Pipelines

Problem DescriptionWhen running an SDC pipeline which processes records with Strings containing UTF-8 special characters, these special characters are being replaced by question marks (? or �) in various parts of your pipeline.ExampleInput record:{"productName": "My Product™"}Output record:{"productName": "My Product�"}Root CauseThis problem indicates that your Java Runtime Environment is using a non-UTF-8 character set which does not support these special characters, so the special characters are being replaced by the � (U+FFFD) REPLACEMENT CHARACTER.By default, SDC will try to set your JAVA_OPTS to use the UTF-8 encoding, however it is possible that the JAVA_OPTS parameters are being set outside of SDC and are overriding either of the following JVM parameters which tells the JRE what encoding to use at runtime: file.encoding and sun.jnu.encoding.The JRE also has a default character set which it will use if these two parameters are not specified. This default character set can vary be

86781
D
Roadie
10 months ago
Douglas R
Fan
Douglas RFan
posted in Community Articles and Got a Question?
"Failed at Step EXEC - Permission denied" when starting SDC as a Service on Systems with SELinux

"Failed at step EXEC  - Permission denied" when starting SDC as a Service on Systems with SELinux ProblemWhen starting StreamSets Data Collector as a service under systemd, the service fails immediately on startup.  The following error is shown when the status of the service is checked (using the systemctl status sdc command).systemd[13801]: sdc.service: Failed at step EXEC spawning /opt/streamsets-datacollector/bin/streamsets: Permission denied CauseThis error indicates that the script SDC_HOME/bin/streamsets (which is used to start SDC) could not be launched by systemd.   This can be caused by incorrect file or directory permissions or by problems with the SELinux context (on systems where SELinux is enabled). SolutionStep 1:  Examine the Service ConfigurationBefore looking at possible permissions issues, it is important to verify that the user, group, and installation directory for the service are correctly configured in the service unit file, /etc/systemd/system/sdc.service.  Exami

87521
devops18
Fan
10 months ago
mfalcon
mfalconNew Member
asked in Community Articles and Got a Question?
Export Job error

When I’m trying to export a job it shows the following response{"ISSUES":[{"code":"CONNECTION_IMPORT_EXPORT_02","message":"Failed to export connections metadata, error is Issues: [CONNECTION_IMPORT_EXPORT_01 - Cannot retrieve the results from the connections application (HTTP Code 400)]"}]}'It’s only hapenning with jobs that doens’t have any data on the Parameters tab of Configuration section.Does anybody now why is happening this?

400
mfalcon
11 months ago
san
Fan
sanFan
asked in Community Articles and Got a Question?
StreamSet with MS SQL connection Issues

Hi The Top Contributors, I have tried many connections on SQL with StreamSet but it is not workingsRemediation steps:Make sure the connection properties have been properly set to allow SSL connections with the database. Check out the links below for more information on how to configure the SQL Server properties. - https://learn.microsoft.com/en-us/sql/connect/jdbc/connecting-with-ssl-encryption?view=sql-server-ver15#configuring-the-connection - https://learn.microsoft.com/en-us/sql/connect/jdbc/setting-the-connection-properties?view=sql-server-ver15 Technical details:The SSL negotiation has failed and the driver found has not been able to initialize the Connection Pool: Failed to initialize pool: "encrypt" property is set to "true" and "trustServerCertificate" property is set to "false" but the driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption: Error: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderEx

2002
antmcmullen
StreamSets Employee
11 months ago
aark
aarkNew Member
asked in Community Articles and Got a Question?
Pagination to a streamsets pipeline throws null pointer exception

Trying to add pagination to a streamsets pipeline that requests data from Tableau. Currently, the API is of the format: ttps://us-west-2b.online.tableau.com/api/3.3/sites/${record:value('/credentials/site/id')}/workbooks?pageNumber=${startAt} , where startAt starts at 1 and should go to 4. I selected the "By Page Number" pagination mode option with an offset of 1. However, I am getting a null pointer exception on making the request. 

130
aark
11 months ago
J
Opening Band
jyuanOpening Band
asked in Community Articles and Got a Question?
Can the temporary directories be places in another location

We are running the SDC on Linux servers. I noticed that many temporary directories are created under the /tmp directory. I assume that this is the default location for these sub-directories. However, I would like to use a different location for them. For example, maybe I can use /data/tmp?What’s the procedure to specify another location for those temporary sub-directories?Thanks!

564
J
Opening Band
11 months ago

Badge winners

Show all badges
Terms & Conditions