Skip to main content

StreamSets Platform

Get inspired and gain all the knowledge you need

1199 Topics
S
Roadie
shivrajRoadie
asked in Community Articles and Got a Question?
Release connection Port of vm instace, used by pipeline in which Http server origin is used

I have developed a pipeline that utilizes an Http Server as the origin. In the configuration of the origin, I have designated port 8000 for the purpose of writing data to the pipeline. However, the issue lies in the fact that the pipeline does not release this port. I aim to utilize the same port for multiple pipelines, distinguishing them by their respective Application-Id. Upon executing the initial pipeline, the port remains occupied and fails to release. Upon attempting to execute a second pipeline using the same origin, I encountered an error message stating "Port is not free."If anyone got solution, Please Help me out with this.

282
S
Roadie
7 months ago
Hamid K
Fan
Hamid KFan
asked in Community Articles and Got a Question?
JSON payload validation using json schema

Hello community, I am vuilding a pipeline that receives http requests providing a JSON payload in the body.I want to validate the payload against a json schema. Now I have come accross this article which is interesting https://streamsets.com/blog/json-validator/. I keep getting this error : JSON_VAL_02 - The JSON object supplied is invalid: org.json.JSONException: Expected a ':' after a key at 6 [character 7 line 1] When I check the record as json string checkbox, it raises the exception … com.streamsets.pipeline.api.impl.TypeSupportConversionException: Cannot convert Map to com.streamsets.pipeline.api.impl.StringTypeSupport@374848aAlso with that processor component I am restricted as I would like to fetch the json schema from a repository.I have tried using the javascript evaluator as well and also there I am running into trouble with library versions I believe …Below the error I am getting …Caused by: java.io.IOException: resource /draftv4/schema not foundUsing libraries …json-schem

1410
Hamid K
Fan
7 months ago
Hamid K
Fan
Hamid KFan
asked in Community Articles and Got a Question?
Testing HTTP server origin on SDC deployed in docker

Hello all,I am trying to run a pipeline with an HTTP server origin to receive messages and then push them to a kafka topic. When I start my docker container, the port is not accessible on my local laptop when I don’t enable port-forwarding for the port the origin is listening on. But when I enable port-forwarding then I get the notification : “The engine cannot be reached. Check that the engine is running and can communicate with Control Hub.” Can anyone help me out on this?Many thanks!Hamid K.

351
Hamid K
Fan
8 months ago
F
Fan
fhermosillaFan
asked in Community Articles and Got a Question?
Using GCP EL credentials in a custom function

Hi Everyone.We are working in a solution which integrates GCP Secret Manager and Streamsets.The client stores a secret in JSON format, and wants to extract a field with his data. Is similar to this format:   How can I extract the dbuser field??I now there are custom functions, but don’t know if I can use the EL function from Streamsets to obtain the JSON info.Can I use the ${credential:get("gcp", "group@org", "dbuser?latest") as an argument? so the function returns only the dbuser value. Thanks! 

140
F
Fan
8 months ago
hrishikeh132609
Fan
hrishikeh132609Fan
asked in Community Articles and Got a Question?
Teradata to Snowflake (Extract and Load)

Hi,Use case.We are load data from Teradata to Snowflake.as we not doing any transformation just extract the Data and loading in Snowflake.which engine will be the best for this use case.1.Data collector​​​​​​​2.Transformer for spark.Note: we have data in millions  Best regards,Hrishikesh 

150
hrishikeh132609
Fan
8 months ago
N
Fan
nachiket_petheFan
asked in Community Articles and Got a Question?
After enabling https on Control Hub, login screen redirects back to login screen.

We have performed following steps according to this documentation - https://docs.streamsets.com/portal/controlhub/latest/onpremhelp/controlhub/UserGuide/Install/EnableHTTPS.html1. stored .p12 file in etc/dpm directory2. made changes in dpm.properties (exactly as given in the documentation)3. made changes in common-to-all.properties (exactly as given in the documentation)4. stored keystore password in keystore-password.txt5. start the control hub manuallywe are using control hub version 3.51.4

170
N
Fan
8 months ago
pedro
StreamSets Employee
pedroStreamSets Employee
posted in Community Articles and Got a Question?
How to use Content Type ‘multipart/form-data’ in the HTTP Client stage

SDC’s HTTP Client stage does not have a native option to send HTTP requests using the Content Type ‘multipart/form-data’. Using the default options will result in SDC making the call seemingly correctly but it will not add the required boundary characters that a request using this kind of Content Type requires. However, we can force SDC to send the call in the correct multipart/form-data’ format. To do this you will need to set these parameters in the following way:*** For this example, the request SDC would make is equivalent to the following curl command:curl --location 'https://test.url.com/api/jobs' \--header 'Accept: application/json' \--header 'Authorization: Bearer *********************' \--header 'Cookie: staging-connections-canary="45412ba154f512d34"' \--form 'options="{\"options\":{},\"url\":\"https://test.url.com\",\"sslVersion\":\"TLS 1.2\",\"password\":xxxxxxxxxx,\"userId\":xxxxxxxxxx,\"displayName\":\"Pedro\",\"jobName\":\"pedro_test_job\",\"batchsize\":1000,\"waitTimeout

13241
N
Opening Band
8 months ago
N
Opening Band
N_aliOpening Band
asked in Community Articles and Got a Question?
Move Files from sftp server to httpclient dst as an attachment?

Hi , i need to transfer csv files from an sftp server to post it as an attachment in a post API (http client),Can you help me figure out how to accomplish this ?

70
N
Opening Band
8 months ago
Pradeep
StreamSets Employee
PradeepStreamSets Employee
asked in Community Articles and Got a Question?
API which returns all pipelines

Problem description: By default https://<control-hub or platform endpoint>:<port-number>/pipelinestore/rest/v1/pipelines returns the pipelines which are committed. Pipeline count which you note in below image is the number committed pipelines. What if we want the pipelines api to return non-committed or draft pipelines too?

201
Pradeep
StreamSets Employee
8 months ago
john.mcavoy
StreamSets Employee
john.mcavoyStreamSets Employee
posted in Community Articles and Got a Question?
UTF-8 Special Characters being replaced by Question Marks (�)in Pipelines

Problem DescriptionWhen running an SDC pipeline which processes records with Strings containing UTF-8 special characters, these special characters are being replaced by question marks (? or �) in various parts of your pipeline.ExampleInput record:{"productName": "My Product™"}Output record:{"productName": "My Product�"}Root CauseThis problem indicates that your Java Runtime Environment is using a non-UTF-8 character set which does not support these special characters, so the special characters are being replaced by the � (U+FFFD) REPLACEMENT CHARACTER.By default, SDC will try to set your JAVA_OPTS to use the UTF-8 encoding, however it is possible that the JAVA_OPTS parameters are being set outside of SDC and are overriding either of the following JVM parameters which tells the JRE what encoding to use at runtime: file.encoding and sun.jnu.encoding.The JRE also has a default character set which it will use if these two parameters are not specified. This default character set can vary be

81751
D
Roadie
8 months ago
Douglas R
Fan
Douglas RFan
posted in Community Articles and Got a Question?
"Failed at Step EXEC - Permission denied" when starting SDC as a Service on Systems with SELinux

"Failed at step EXEC  - Permission denied" when starting SDC as a Service on Systems with SELinux ProblemWhen starting StreamSets Data Collector as a service under systemd, the service fails immediately on startup.  The following error is shown when the status of the service is checked (using the systemctl status sdc command).systemd[13801]: sdc.service: Failed at step EXEC spawning /opt/streamsets-datacollector/bin/streamsets: Permission denied CauseThis error indicates that the script SDC_HOME/bin/streamsets (which is used to start SDC) could not be launched by systemd.   This can be caused by incorrect file or directory permissions or by problems with the SELinux context (on systems where SELinux is enabled). SolutionStep 1:  Examine the Service ConfigurationBefore looking at possible permissions issues, it is important to verify that the user, group, and installation directory for the service are correctly configured in the service unit file, /etc/systemd/system/sdc.service.  Exami

78761
devops18
Fan
9 months ago
mfalcon
mfalconNew Member
asked in Community Articles and Got a Question?
Export Job error

When I’m trying to export a job it shows the following response{"ISSUES":[{"code":"CONNECTION_IMPORT_EXPORT_02","message":"Failed to export connections metadata, error is Issues: [CONNECTION_IMPORT_EXPORT_01 - Cannot retrieve the results from the connections application (HTTP Code 400)]"}]}'It’s only hapenning with jobs that doens’t have any data on the Parameters tab of Configuration section.Does anybody now why is happening this?

400
mfalcon
9 months ago
san
Fan
sanFan
asked in Community Articles and Got a Question?
StreamSet with MS SQL connection Issues

Hi The Top Contributors, I have tried many connections on SQL with StreamSet but it is not workingsRemediation steps:Make sure the connection properties have been properly set to allow SSL connections with the database. Check out the links below for more information on how to configure the SQL Server properties. - https://learn.microsoft.com/en-us/sql/connect/jdbc/connecting-with-ssl-encryption?view=sql-server-ver15#configuring-the-connection - https://learn.microsoft.com/en-us/sql/connect/jdbc/setting-the-connection-properties?view=sql-server-ver15 Technical details:The SSL negotiation has failed and the driver found has not been able to initialize the Connection Pool: Failed to initialize pool: "encrypt" property is set to "true" and "trustServerCertificate" property is set to "false" but the driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption: Error: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderEx

1971
antmcmullen
StreamSets Employee
9 months ago
I
Fan
ingridmirnikFan
asked in StreamSets Academy
Why does self-managed server keep crashing on engines restart and what are proper transfomer configurations?

We‘ve set up a machine for Self-Managed Deployments (specifications below) with Docker as it’s said in the documentation. Machine specifications:Operating system: Linux (ubuntu 22.04) Disk size: 30GiB RAM: 4GiB Performance tier: P10 - 500 IOPS, 100MMB/sWe succesfully created some Data Collector and Transfomer engines. The problem occurs when we try to restart/run all of the engines at the same time. Server just freezes and eventually stops responding.If you look at the metrics of the machine, you can see that it stops usually when “Disk Bytes” peek. My first question is, what could cause this huge usage of resources to crash the server eventually? The next things are Transformer engines. In documentation it says:“Important: When using a Transformer engine that works with a Spark cluster, edit the transformer.base.http.url property on the Transformer Configuration tab. Uncomment the property and set it to the Transformer URL. For more information, see Granting the Spark Cluster Access t

240
I
Fan
9 months ago
aark
aarkNew Member
asked in Community Articles and Got a Question?
Pagination to a streamsets pipeline throws null pointer exception

Trying to add pagination to a streamsets pipeline that requests data from Tableau. Currently, the API is of the format: ttps://us-west-2b.online.tableau.com/api/3.3/sites/${record:value('/credentials/site/id')}/workbooks?pageNumber=${startAt} , where startAt starts at 1 and should go to 4. I selected the "By Page Number" pagination mode option with an offset of 1. However, I am getting a null pointer exception on making the request. 

130
aark
9 months ago
J
Opening Band
jyuanOpening Band
asked in Community Articles and Got a Question?
Can the temporary directories be places in another location

We are running the SDC on Linux servers. I noticed that many temporary directories are created under the /tmp directory. I assume that this is the default location for these sub-directories. However, I would like to use a different location for them. For example, maybe I can use /data/tmp?What’s the procedure to specify another location for those temporary sub-directories?Thanks!

564
J
Opening Band
9 months ago
tejas_surve
Roadie
tejas_surveRoadie
asked in Community Articles and Got a Question?
How to use Azure Service Bus ?

How to use Azure service Bus as messaging queue in StreamSets?How to configure Connection?

320
tejas_surve
Roadie
9 months ago
S
Roadie
shivrajRoadie
asked in Community Articles and Got a Question?
How to connect Hazelcast MQ with Streamsets Pipeline?

I want to establish a connect between Hazelcast-MQ to use it as origin and destination. Couldn’t find any information about that in Streamsets Documentation. Please help me out with this.

331
Dimas Cabré i Chacón
StreamSets Employee
9 months ago
Francisco
Fan
FranciscoFan
asked in Community Articles and Got a Question?
JDBC Query Consumer SSL error

Hello all, I had a pipeline that was running well with a JDBC Query Consumer stage with a simple select to a MSSQL database server, until one day I started getting the following error:DBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: "Certificates do not conform to algorithm constraints". ClientConnectionId:1115d9c7-5599-411e-ba8e-3c379d2ac0ac: hikariConfigBean.connectionString Is this related to changes that may have been made in the database server? There is no TLS configuration on this stage, I’m not sure if it’s using TLS by default. Checking with the database owner, he indicated that the SQL Server is compatible with TLS 1.2. Any help is appreciated, thank you.

600
Francisco
Fan
9 months ago
bonthunagireddy03
Fan
bonthunagireddy03Fan
asked in Community Articles and Got a Question?
How to connect to IBM MQ using SDC JMS origin

Hi Team,we are trying to establish a connection to IBM MQ using JMS origin, but we couldn’t be able to do as could not find much information on StreamSets documentation (https://docs.streamsets.com/portal/#datacollector/latest/help/datacollector/UserGuide/Origins/JMS.html?hl=jms). Do you have any reference document which says how to establish a connection to IBM MQ? much thanks in advance!!

921
Kartik Medavarapu
Roadie
9 months ago
john.mcavoy
StreamSets Employee
john.mcavoyStreamSets Employee
asked in Community Articles and Got a Question?
How to Cleanup Job History for SDC and SCH

ProblemOld SCH job run histories are taking up a lot of disk space either in SDC nodes or in SCH’s DB, and these histories need to be limited for better performance. SolutionJob run history is stored in SCH and in SDC (if SDC has configured dpm.runHistory.enabled=true in $SDC_CONF/dpm.properties).In SCH, you can configure Maximum number of job runs for each Organization. This is the max number of runs retained for each job's history SCH periodically runs a clean-up task to purge all old job histories when this limit is reached. SDC can also retain a local copy of job run histories if dpm.runHistory.enabled=true is configured in $SDC_CONF/dpm.properties. SDC keeps runtime information pertaining to each running pipeline in $SDC_DATA/runInfo. SDC deletes any local job history information in $SDC_DATA/runInfo whenever a pipeline finishes. If you set dpm.runHistory.enabled=true, then SDC will copy each job’s history from $SDC_DATA/runInfo into $SDC_DATA/runHistory before this clean-up oc

630
john.mcavoy
StreamSets Employee
9 months ago
som
Fan
somFan
posted in Community Articles and Got a Question?
Socket Timeout Exceptions when using "Start Job" stage.

Environment:* StreamSets Control Hub 3.19.2* SDC server 5.2.0* Transformer 4.3.0-0049ISSUE:Start Jobs stage failing in 5.2.0 with: java.net.SocketTimeoutException: Read timed outSOLUTION:Data Collector v5.2.0 sets a default read timeout of 10 seconds.Control Hub logs show numerous jobs take more than 10 seconds to start, with the highest being 42 seconds.Customer to update read timeout setting to a higher value, such as 60  or 90 seconds to fix this issue in this scenario.Click on the Start Jobs stage Click on the Configuration in the left menu Click on the HTTP tab Change Read Timeout (ms) from 10000 to another value, such as 60000 or 90000

430
som
Fan
9 months ago
som
Fan
somFan
posted in Community Articles and Got a Question?
ORACLE_08 - Oracle driver not found.

Oracle driver not found error can happen because of many reasons .Below are few of the scenario. Please check if the external library is placed in the correct directory.it should be in “/opt/streamsets-datacollector/streamsets-libs-extras/streamsets-datacollector-jdbc-lib/lib/”.In case this path does not exist, create it. We recommend to install external drivers from the same SDC UI  by going to your pipeline> Select the stage > External libraries > Install External Libraries and then restart your SDC. Please check if there are multiple jdbc jars with different versions are in use for different libraries as this could add some issue on how Linux distributes the jars. Linux gives back the drivers in a non-deterministic order, and it can lead to this situation in very rare cases.

60
som
Fan
9 months ago
aysudy
Fan
aysudyFan
asked in StreamSets Academy
Cannot use Strigo / This course has ended

Hello,  I was doing the StreamSets fundamentals training and I have got this notification from Strigo, saying “This course has ended”. Actually It hasn’t ended yet. How can I reaccess the account? I am Software AG employee. 

462
aysudy
Fan
9 months ago
Pradeep
StreamSets Employee
PradeepStreamSets Employee
asked in Community Articles and Got a Question?
JdbcQueryExecutor - Can't execute querynet.snowflake.client.jdbc.SnowflakeSQLException

How to solve the below error in SDC logs faced by pipeline(s)?JdbcQueryExecutor - Can't execute querynet.snowflake.client.jdbc.SnowflakeSQLException: Execution error in store procedure PROC_AUTO_MERGE:Statement '01b0b234-0504-da19-0032-6b8704c17d96' has locked table 'TEST_TABLE' in transaction 1701430332198000000 and this lock has not yet been released.Your statement '01b0b234-0504-d944-0032-6b8704c16e32' was aborted because the number of waiters for this lock exceeds the 20 statements limit. At Statement.execute, line 155 position 17 

1161
Pradeep
StreamSets Employee
9 months ago

Badge winners

Show all badges
Terms & Conditions