StreamSets Platform

Issue:-HTTP Client pipeline throws following error after running for few hours:-HTTP_40 - Error sending resource. HTTP-Status: 401 Reason: SubCode=40106: Access Control Serivce token supplied for this operation has expired. Retry the operation. Resolution:-The Azure AD tokens expire at 60 minutes. These parameters were set to 56 minutes. This change caused the SDC to terminate the connection under the 60 minutes where the token is valid and forced it to renew the token.After increasing Connect Timeout and Read Timeout in HTTP Client to => 60 minutes the issue was resolved.

Issue:-While scheduler trying to start the job on scheduled time it failed with following error:-ERROR InfluxDBTimeSeriesBackend - Error writing metrics to sub systems: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@ef0b00f rejected from java.util.concurrent.ThreadPoolExecutor@282e5632[Running, pool size = 4, active threads = 4, queued tasks = 10000, completed tasks = 12368917] Resolution:-Influx DB is not processing the requests efficiently as the queued tasks reached it limit to 10000. There can be two approaches to address the issue:-Increase the task limit using following property in the timeseries-app.properties.  metrics.callable.queue.size=20000       2. Get in touch with InfluxDB support to optimise the performance.

Issue:-SQL server CDC is failing with the following error:-JDBC_75 - Jdbc Runner Failed. Reason java.util.concurrent.ExecutionException: java.lang.NullPointerException: value cannot be null Resolution:-The CDC was enabled at the table level instead of the DataBase level. After enabling the CDC at the DataBase level, the NULL pointer issue was resolved.

Issue:-Users can grep the logs for the specific jobs from sdc.log, however, for some jobs the logs can’t be accessed from the UI.Resolution:-The name of the pipeline started with '?'. (Special Character), after removing the special character from the beginning of the pipeline logs can be accessed from the UI. 

Issue:-While configuring the JKS for SDC, following error has been observed:-keytool error: java.io.IOException: Integrity check failed: java.security.UnrecoverableKeyException: Failed PKCS12 integrity checkingResolution:-The error integrity check failed may occur due to an incorrect key password. A new PKCS12 store has been created and imported the certificates to the newly created PKCS12

Rotation of GC logging can be enabled using following parameter:- vi $SDC_DIST/libexec/sdc-env.sh/ Add -Xloggc:/PATH/TO/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=300M to export SDC_JAVA_OPTS lineFollowing is the test configuration :-export SDC_JAVA_OPTS="-Xmx1024m -Xms1024m -server -XX:-OmitStackTraceInFastThrow ${SDC_JAVA_OPTS} -Dlog4j2.formatMsgNoLookups=true -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/Users/ajinkya/utilities/streamsets-datacollector-4.2.1/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=300M" 

Issue:-Oracle CDC logs shows incorrect transaction id.Transaction id shown:-"Removing transaction with id: 2022-11-28T14:34:48"Expected Transaction id:-"Removing transaction with id: 2.2.531560" Resolution:-This is caused by a regression which has been resolved in version SDC 5.4.0.

http client is not working for data collector producing below error:-com.streamsets.pipeline.api.StageException: HTTP_32 - Error executing request. HTTP-Status: NULL Reason: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested targeteven for simple rest api like https://dummyjson.com/products/want to know why this is happening?

The Issue:  You are experiencing intermittent login issues when LDAP is in use, even when passwords have not been changed. You may also notice the following errors in the logs:  WARN JpaSecurityBackend - User ‘<username>’ failed to login: [SECURITY_34  - Invalid password]   ERROR BlockingConnectionPool ... {truncated} ... unable to connect to the ldaporg.ldaptive.provider.ConnectionException:javax.naming.CommunicationException: … {truncated} ...  [Root exception is java.net.SocketTimeoutException: connect timed out]    The Solution:  One of the potential causes for this issue is a socket timeout or network latency. If you are using Control Hub On-premises, there are a few related configurations concerning LDAP which can be changed in order to remedy this issue. To change these settings:  On the command line, navigate to the $DPM_CONF directory (usually in /etc) and open the security-app.properties file with a text editor.  Find the following two properties in the configuration

In StreamSets Data Collector, the SQL Server CDC origin has a Queries per Second setting which defaults to 10. This setting allows you to control how much traffic is sent to SQL Server to avoid overwhelming it.In some circumstances, having a limit on the number of queries per second can cause the pipeline to fall behind on CDC changes. In extreme cases, the pipeline might appear to be receiving no records at all.If you encounter this, you can try increasing the setting, or you can set it to 0 which means no limit. Bear in mind the capabilities of the server when changing this setting. If the server shows signs of being strained or overwhelmed by Data Collector’s queries, we recommend trying to find a good upper bound for this setting to strike a balance between receiving timely updates and causing stress on the server.

Environment: StreamSets Data Collector v5.7.2 Any ControlHub version.  Issue: When you installed Kafka 2.7 on the data collector version 5.7.2, and it is currently functioning without any issues. However, when attempting to upgrade the library to version 3.5 and subsequently removing older version 2.7, 2,5, 2.6 the ControlHub UI displays a validation error, as expected. The problem arises when attempting to use the Click here to update button in the DPM-UI, as it does not seem to work as intended. Keep asking users to update the libraries.Here is the screenshots of the issue:-Update the stage LibraryStill failingSolution:Upon investigation, we found that we are hitting a UI bug and will be fixed in DPM-19532. As a workaround, you can keep both libraries installed temporarily during the upgrade and use the stage library selector in the stage configuration.Keep both the libraries and select the latest ones.Publish the pipeline and it will work. 

 Thread starvation or clock leap can be caused  due to different causes but mainly it happens because of below reasons-High load on the engine The server is so busy that thread starvation is occurring, which is preventing the housekeeper thread from running. GC running for a long time. Insufficient resources (heap or CPU).Possible solutions for this issue could be the following: Increasing the memory of the engine.  Rebalancing the jobs running on the engine.  Dividing a big pipeline into multiple parts. Reducing the Batch size

 Currently, JVM (openjdk/Oracle) only exposes overall CPU or memory metrics for a process.As of now, it's not possible at jvm level to pull the individual pipeline memory details.To know more about the the memory consumption we can analyse heap dumps ,thread dumps  and gc.log when the issue is happening. heap dumps:sudo -u sdc jcmd <pid>  GC.heap_dump <filename>.hprof Thread dumps:for i in {0..5} ; do echo $i ; jstack {pid} > /tmp/sdc-$(date +%Y-%m-%d.%H:%M:%S).jstack ; sleep 5 ; done

In DataOps platform, the advanced engine configuration files are visible in deployment UI and can be read using SDK as well.However, I want to read advanced engine configurations: log4j, proxy, security policy and java configuration files from engine that is registered with control hub 3.x using SDK 3.x. One of the attributes of sdc instance is “sdc_configuration”, but it only contains parameters related to “sdc.properties” and “credentialStore” property files, not the other configuration files. I have attached the engine configuration available on DataOps platform UI. 

Is it possible to get the running job’s metric using Python sdk, e.g execution time, status etc?

I am trying to regenerate api credentials and store them in some location(text file). I tried it in many ways but seems to be not working 

from streamsets.sdk import ControlHubfrom streamsets.sdk import DataCollectorfrom streamsets.sdk import sch_modelsimport jsonDataCollector.VERIFY_SSL_CERTIFICATES = False###Add the sdc details of your environmentsdc = DataCollector('http://localhost:18650/', username='admin', password='admin')pipeline_json = sdc.export_pipeline(pipeline=sdc.pipelines.get(title='stop test'), include_library_definitions=True, include_plain_text_credentials=True)with open('./from_sdc_for_sdc.json', 'w') as f: json.dump(pipeline_json, f) 

ISSUE:We have two engines in one deployment and wanted to run one engine in http and other in https.Solution:It is not possible to have different configuration for the single deployment. Configuration is tied to each deployments and what ever change we do ,it will be done for all the available engines on that deployments.We normally use multiple engines as part of single deployment when the purpose is to keep same configuration across all data collector. In this case, we can  create separate deployment for both the engines.

Environment:  Transformer v5.4. Issue: It has been observed that following errors are appearing in the transformer logs without any pipeline running on v5.4.2023-08-14 04:04:35,552 [user:] [pipeline:] [stage:] WARN Reflections - could not get type for name com.sun.xml.txw2.output.CharacterEscapeHandler from any class loaderst.shaded.org.reflections.ReflectionsException: could not get type for name com.sun.xml.txw2.output.CharacterEscapeHandler at st.shaded.org.reflections.ReflectionUtils.forName(ReflectionUtils.java:312) ~[streamsets-transformer-dist_2.11-5.4.0.jar:?] at st.shaded.org.reflections.Reflections.expandSuperTypes(Reflections.java:382) ~[streamsets-transformer-dist_2.11-5.4.0.jar:?] at st.shaded.org.reflections.Reflections.<init>(Reflections.java:140) ~[streamsets-transformer-dist_2.11-5.4.0.jar:?] at st.shaded.io.swagger.jaxrs.config.BeanConfig.classes(BeanConfig.java:276) ~[streamsets-transformer-dist_2.11-5.4.0.jar:?] at st.shaded.io.swagger.jaxrs.config.BeanConfig

ISSUE:We have 2 data collectors(sdc1 & sdc2) on different hosts configured with the same label(prod234) . This morning control hub was routing all jobs with job label prod234 to a single data collector (sdc2) though sdc1 was never failed. Troubleshooting:We have found the Issue is likely related to the default values for cpu (80%), memory (100%) and max pipelines (10K) which was set in the engine configuration.The running jobs are memory intensive and allocating new jobs to a collector that's even close to 100% is asking for trouble.We've adjusted the thresholds to cpu (80%), memory (75%) and max pipelines (10) and the issue have never reappeared.

ISSUE:We have 2 data collectors(sdc1 & sdc2) on different hosts configured with the same label(prod234) . This morning control hub was routing all jobs with job label prod234 to a single data collector (sdc2) though sdc1 was never failed. Troubleshooting:We have found the Issue is likely related to the default values for cpu (80%), memory (100%) and max pipelines (10K) which was set in the engine configuration.The running jobs are memory intensive and allocating new jobs to a collector that's even close to 100% is asking for trouble.We've adjusted the thresholds to cpu (80%), memory (75%) and max pipelines (10) and the issue have never reappeared.

Environment:StreamSets Data Collector (SDC) 5.8.0  Pipelines which use the Oracle CDC stageIssue: In SDC 5.8.0, pipelines using the Oracle CDC stage will encounter an issue where the following error is produced: ORACLE_CDC_1137 - Unexpected operation received from LogMiner:<output truncated>Resolution: A fix is planned for the upcoming SDC 5.8.1 release. 

We are going to configure the Subscriptions with the free webhook (https://webhook.site/) tool. A webhook is a user-defined HTTP callback - an HTTP request that the pipeline sends automatically when certain actions occur. You can use webhooks to automatically trigger external tasks based on an HTTP request. Tasks can be as simple as sending a message through an application API or as powerful as passing commands to the Data Collector command line interface. [1]. https://docs.streamsets.com/portal/controlhub/latest/help/datacollector/UserGuide/Pipeline_Configuration/Webhooks.html?hl=webhook Step1: Setting up a test pipeline DevDataGen ==> Trash. Step2: Reconfigure the control-hub subscription to activate when there is a change in the status of the Job. You can open the https://webhook.site/ and you will find the URL details. Here is my test subscription configurations: Step3: Start the pipeline and you will see subscription is getting triggered. You can confirm the same on the Webhook

BackgroundThe TEXT Data Format parser breaks-up text files containing multiple lines into individual records for each line. Unlike the HTTP Client Origin, the HTTP Client Processor creates one HTTP request for every input record, and only outputs one record per HTTP response. This means the HTTP Processor will only return the first record if the HTTP Response data is split into multiple records which can cause issues when trying to read multi-line HTTP Responses. SolutionIn order to read multi-line TEXT responses via the HTTP Processor, set the Data Format to BINARY which will read the entire HTTP Response as a single BYTE_ARRAY record. The BYTE_ARRAY data can then be converted into a String using the Field Converter Processor to output. 

Environment:Streamsets Data Collector v4.2.1. Kafka Multi-topic consumer origin. Issue:It has been observed that Kafka origin is failing with the following error message when “produce single records” configuration is on + data is in JSON format.2021-01-10 15:31:12,193 [user:test] [pipeline:test_kafka_working] [runner:] [thread:Kafka pipeline-thread1] [stage:KafkaMultitopicConsumer_01] INFO MultiKafkaSource - Multi kafka thread halted unexpectedly: KAFKA_29 - Error fetching data from Kafka: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0java.util.concurrent.ExecutionException: com.streamsets.pipeline.api.StageException: KAFKA_29 - Error fetching data from Kafka: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.streamsets.pipeline.stage.origin.multikafka.MultiKafkaSource.produce(MultiKafkaSource.java:163) at com.streamsets.pipeline.ap