StreamSets Platform

Hi , i need to transfer csv files from an sftp server to post it as an attachment in a post API (http client),Can you help me figure out how to accomplish this ?

Problem description: By default https://<control-hub or platform endpoint>:<port-number>/pipelinestore/rest/v1/pipelines returns the pipelines which are committed. Pipeline count which you note in below image is the number committed pipelines. What if we want the pipelines api to return non-committed or draft pipelines too?

Problem Description When running an SDC pipeline which processes records with Strings containing UTF-8 special characters, these special characters are being replaced by question marks (? or �) in various parts of your pipeline. Example Input record: {"productName": "My Product™"} Output record: {"productName": "My Product�"} Root Cause This problem indicates that your Java Runtime Environment is using a non-UTF-8 character set which does not support these special characters, so the special characters are being replaced by the � (U+FFFD) REPLACEMENT CHARACTER. By default, SDC will try to set your JAVA_OPTS to use the UTF-8 encoding, however it is possible that the JAVA_OPTS parameters are being set outside of SDC and are overriding either of the following JVM parameters which tells the JRE what encoding to use at runtime: file.encoding and sun.jnu.encoding. The JRE also has a default character set which it will use if these two parameters are not specified. This default character set c

"Failed at step EXEC - Permission denied" when starting SDC as a Service on Systems with SELinux Problem When starting StreamSets Data Collector as a service under systemd, the service fails immediately on startup. The following error is shown when the status of the service is checked (using the systemctl status sdc command). systemd[13801]: sdc.service: Failed at step EXEC spawning /opt/streamsets-datacollector/bin/streamsets: Permission denied Cause This error indicates that the script SDC_HOME/bin/streamsets (which is used to start SDC) could not be launched by systemd. This can be caused by incorrect file or directory permissions or by problems with the SELinux context (on systems where SELinux is enabled). Solution Step 1: Examine the Service Configuration Before looking at possible permissions issues, it is important to verify that the user, group, and installation directory for the service are correctly configured in the service unit file, /etc/systemd/system/sdc.service. Examin

When I’m trying to export a job it shows the following response{"ISSUES":[{"code":"CONNECTION_IMPORT_EXPORT_02","message":"Failed to export connections metadata, error is Issues: [CONNECTION_IMPORT_EXPORT_01 - Cannot retrieve the results from the connections application (HTTP Code 400)]"}]}' It’s only hapenning with jobs that doens’t have any data on the Parameters tab of Configuration section. Does anybody now why is happening this?

Hi The Top Contributors, I have tried many connections on SQL with StreamSet but it is not workings Remediation steps: Make sure the connection properties have been properly set to allow SSL connections with the database. Check out the links below for more information on how to configure the SQL Server properties. - https://learn.microsoft.com/en-us/sql/connect/jdbc/connecting-with-ssl-encryption?view=sql-server-ver15#configuring-the-connection - https://learn.microsoft.com/en-us/sql/connect/jdbc/setting-the-connection-properties?view=sql-server-ver15 Technical details: The SSL negotiation has failed and the driver found has not been able to initialize the Connection Pool: Failed to initialize pool: "encrypt" property is set to "true" and "trustServerCertificate" property is set to "false" but the driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption: Error: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilde

We‘ve set up a machine for Self-Managed Deployments (specifications below) with Docker as it’s said in the documentation. Machine specifications: Operating system: Linux (ubuntu 22.04) Disk size: 30GiB RAM: 4GiB Performance tier: P10 - 500 IOPS, 100MMB/s We succesfully created some Data Collector and Transfomer engines. The problem occurs when we try to restart/run all of the engines at the same time. Server just freezes and eventually stops responding. If you look at the metrics of the machine, you can see that it stops usually when “Disk Bytes” peek. My first question is, what could cause this huge usage of resources to crash the server eventually? The next things are Transformer engines. In documentation it says: “Important: When using a Transformer engine that works with a Spark cluster, edit the transformer.base.http.url property on the Transformer Configuration tab. Uncomment the property and set it to the Transformer URL. For more information, see Granting the Spark Cluster Acce

Trying to add pagination to a streamsets pipeline that requests data from Tableau. Currently, the API is of the format: ttps://us-west-2b.online.tableau.com/api/3.3/sites/${record:value('/credentials/site/id')}/workbooks?pageNumber=${startAt} , where startAt starts at 1 and should go to 4. I selected the "By Page Number" pagination mode option with an offset of 1. However, I am getting a null pointer exception on making the request.

We are running the SDC on Linux servers. I noticed that many temporary directories are created under the /tmp directory. I assume that this is the default location for these sub-directories. However, I would like to use a different location for them. For example, maybe I can use /data/tmp? What’s the procedure to specify another location for those temporary sub-directories? Thanks!

How to use Azure service Bus as messaging queue in StreamSets?How to configure Connection?

I want to establish a connect between Hazelcast-MQ to use it as origin and destination. Couldn’t find any information about that in Streamsets Documentation. Please help me out with this.

Hello all, I had a pipeline that was running well with a JDBC Query Consumer stage with a simple select to a MSSQL database server, until one day I started getting the following error: DBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: "Certificates do not conform to algorithm constraints". ClientConnectionId:1115d9c7-5599-411e-ba8e-3c379d2ac0ac: hikariConfigBean.connectionString Is this related to changes that may have been made in the database server? There is no TLS configuration on this stage, I’m not sure if it’s using TLS by default. Checking with the database owner, he indicated that the SQL Server is compatible with TLS 1.2. Any help is appreciated, thank you.

Hi Team, we are trying to establish a connection to IBM MQ using JMS origin, but we couldn’t be able to do as could not find much information on StreamSets documentation (https://docs.streamsets.com/portal/#datacollector/latest/help/datacollector/UserGuide/Origins/JMS.html?hl=jms). Do you have any reference document which says how to establish a connection to IBM MQ? much thanks in advance!!

Problem Old SCH job run histories are taking up a lot of disk space either in SDC nodes or in SCH’s DB, and these histories need to be limited for better performance. Solution Job run history is stored in SCH and in SDC (if SDC has configured dpm.runHistory.enabled=true in $SDC_CONF/dpm.properties). In SCH, you can configure Maximum number of job runs for each Organization. This is the max number of runs retained for each job's history SCH periodically runs a clean-up task to purge all old job histories when this limit is reached. SDC can also retain a local copy of job run histories if dpm.runHistory.enabled=true is configured in $SDC_CONF/dpm.properties. SDC keeps runtime information pertaining to each running pipeline in $SDC_DATA/runInfo. SDC deletes any local job history information in $SDC_DATA/runInfo whenever a pipeline finishes. If you set dpm.runHistory.enabled=true, then SDC will copy each job’s history from $SDC_DATA/runInfo into $SDC_DATA/runHistory before this clean-up oc

Environment: * StreamSets Control Hub 3.19.2* SDC server 5.2.0* Transformer 4.3.0-0049 ISSUE: Start Jobs stage failing in 5.2.0 with: java.net.SocketTimeoutException: Read timed out SOLUTION: Data Collector v5.2.0 sets a default read timeout of 10 seconds.Control Hub logs show numerous jobs take more than 10 seconds to start, with the highest being 42 seconds. Customer to update read timeout setting to a higher value, such as 60 or 90 seconds to fix this issue in this scenario. Click on the Start Jobs stage Click on the Configuration in the left menu Click on the HTTP tab Change Read Timeout (ms) from 10000 to another value, such as 60000 or 90000

Oracle driver not found error can happen because of many reasons .Below are few of the scenario. Please check if the external library is placed in the correct directory.it should be in “/opt/streamsets-datacollector/streamsets-libs-extras/streamsets-datacollector-jdbc-lib/lib/”.In case this path does not exist, create it. We recommend to install external drivers from the same SDC UI by going to your pipeline> Select the stage > External libraries > Install External Libraries and then restart your SDC. Please check if there are multiple jdbc jars with different versions are in use for different libraries as this could add some issue on how Linux distributes the jars. Linux gives back the drivers in a non-deterministic order, and it can lead to this situation in very rare cases.

Hello, I was doing the StreamSets fundamentals training and I have got this notification from Strigo, saying “This course has ended”. Actually It hasn’t ended yet. How can I reaccess the account? I am Software AG employee.

How to solve the below error in SDC logs faced by pipeline(s)? JdbcQueryExecutor - Can't execute querynet.snowflake.client.jdbc.SnowflakeSQLException: Execution error in store procedure PROC_AUTO_MERGE:Statement '01b0b234-0504-da19-0032-6b8704c17d96' has locked table 'TEST_TABLE' in transaction 1701430332198000000 and this lock has not yet been released.Your statement '01b0b234-0504-d944-0032-6b8704c16e32' was aborted because the number of waiters for this lock exceeds the 20 statements limit. At Statement.execute, line 155 position 17

i have here an sftp origin that i want to pull the files from do some formatting on it and then push it as a bulk post request to HTTP client destination.i keep getting this error 504 gateway timeout with the files with size 200KB or bigger.it is posting the data on the system but returns this error. i tried setting the read timeout configuration and maximum time out values to a much higher values like 15 mins or so in the HTTP client destination but i keep getting the same error though the data is posted in the destination. what should i do to prevent this error?

Control Hub On-Prem component page does not load correctly, yields “Http failure response” error at bottom of page. Problem Opening a Control Hub component app such as Connections or Jobs yields a page which contains only the header of the page, with an error message appearing briefly at the bottom of the page as shown in Figure 1 below. This problem is most likely to be encountered after upgrading Control Hub from an older (before 3.19.0) version. Figure 1: Connections Page not loading properly Cause The component (application) is not being started with Control Hub. Solution Verify that the Control Hub component which fails to load correctly is listed among the apps to start in $DPM_CONF/dpm.properties. In this configuration file, find the apps.to.start property. It should look something like this: apps.to.start=security,pipelinestore,messaging,jobrunner,timeseries,topology,notification,sla,provisioning,scheduler,reporting,sdp_classification,policy,dynamic_preview,connection This prop

Any one can known the Error and how to fix this .

Issue:- While running any pipeline against the EMR cluster pipeline is failing with the following error:- java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;) Resolution:- Use the Transformer version whose scala version matches with EMR cluster.

Issue:- The JMS pipeline is failing with following error:- com.solacesystems.jcsmp.JCSMPErrorResponseException: 503: Transaction Failure: The number of messages in the transaction exceeds the limit [Subcode:63]))' Resolution:- Updated the Solace jars to JMS API 10.11 as the limit is set by the server side. SDC processes the messages handed over by the server.

Issue:- HTTP Client pipeline throws following error after running for few hours:- HTTP_40 - Error sending resource. HTTP-Status: 401 Reason: SubCode=40106: Access Control Serivce token supplied for this operation has expired. Retry the operation. Resolution:- The Azure AD tokens expire at 60 minutes. These parameters were set to 56 minutes. This change caused the SDC to terminate the connection under the 60 minutes where the token is valid and forced it to renew the token. After increasing Connect Timeout and Read Timeout in HTTP Client to => 60 minutes the issue was resolved.

Issue:- While scheduler trying to start the job on scheduled time it failed with following error:- ERROR InfluxDBTimeSeriesBackend - Error writing metrics to sub systems: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@ef0b00f rejected from java.util.concurrent.ThreadPoolExecutor@282e5632[Running, pool size = 4, active threads = 4, queued tasks = 10000, completed tasks = 12368917] Resolution:- Influx DB is not processing the requests efficiently as the queued tasks reached it limit to 10000. There can be two approaches to address the issue:- Increase the task limit using following property in the timeseries-app.properties. metrics.callable.queue.size=20000 2. Get in touch with InfluxDB support to optimise the performance.