- - Knowledge base
Product Updates
Events

30-Day Free Trial: It’s Never Been Easier To Get Started With StreamSets

1 month ago

Home
Community overview
StreamSets Platform
Share your Best Practices

Share your Best Practices

Share your use cases, tips, and or ideas with other StreamSets members

36 Topics
21 Replies

When you subscribe we will email you when there is a new topic in this category

36 Topics

Recently active

Most replies Most views Newest first

LavenderStreamSets Employee

posted in Share your Best Practices

Does SDC support IPv6?

Question: Does SDC support IPv6?Answer: Our Data Collector core language was done in Java and, consequently, supports IPv6. Since we’re on Java Stack, the IPv6 is mostly there for the core Data Collector pieces. For compatibility with Data Collector, it mainly depends on the source / target you want to connect whether it supports IPv6. Since SDC acts as a system integrator, for each stage, we would have to go through it and see if they are IPv6 compatible or not. For example: Oracle DB has started to support IPv6 connectivity since 12c, therefore theoretically Data Collector should also support it.Please note that there are some technologies (and versions) that don’t support IPv6. If the origin or destination doesn’t support it, we currently don't have plans to create workarounds or patches to allow it.Besides that, communications between Data Collector and Control Hub / Platform are done using IPv4, so it should be visible between the engine and the Platform. This is important if you

2 days ago

NCollinsStreamSets Employee

published in Share your Best Practices

ajinkyaStreamSets Employee

asked in Share your Best Practices

CONTAINER_0701 - Stage 'OracleCDC_1' initialization error: java.lang.NullPointerException

Issue:-While using the new Oracle CDC Origin, the following error is encountered:-CONTAINER_0701 - Stage 'OracleCDC_1' initialization error: java.lang.NullPointerExceptionResolution:-Oracle CDC requires following jars to be placed on SDC instance:-ojdbc8.jar orai18n-mapping.jar orai18n.jarReview the section Task:4 Install Required Libraries on the Data Collector in the documentation for more information. Root Cause:-Missing dependency jars.

28 days ago

ajinkyaStreamSets Employee

asked in Share your Best Practices

Why Oracle CDC Origin consuming more than 80% CPU?

Issue:- Oracle CDC Origin is consuming more than 80% CPU while processing lower number of records.Resolution:-Set Buffer Wait Strategy to BLOCKING.Root Cause:-New Oracle CDC Origin comes with new Buffer Wait Strategy options. Busy Spin would use the most amount of the CPU. The strategy can be set as per the environment setup and requirements. Please review the documentation for more information.

28 days ago

LavenderStreamSets Employee

posted in Share your Best Practices

Log in Platform as a local Control Hub user when SAML is enabled

Topic: Org Admin wants to log in Platform as a local Control Hub user when SAML is enabledSolution: In our Platform documentation, when SAML is enabled, it is mentioned that:"After you publish a draft configuration to production, you enable the production configuration to activate it. When enabled, all organization users must log in using SAML authentication.As a user with the Organization Administrator role, you can log in using local or public identity provider authentication to access the Control Hub organization SAML configuration page only."This means that all users must authenticate via SSO SAML, but only Org Admin can log in locally to configure SAML settings when SAML breaks, or to disable SAML. This is different than logging in Control Hub as a local user. In Control Hub, there is a way for users with the Org Admin role or the Control Hub Authentication role to access SAML backdoor and log in Control Hub. Please refer to this article for instructions:

1 month ago

NileshFan

asked in Share your Best Practices

How to pass runtime value to transformer in Snowflake Destination

Hi,I’m trying to merge table data if data is not matched. I have multiple tables, so need to create dynamic pipeline which will take input as Table_Name and Primary Key columns to Merge Configuration. But in runtime, Snowflake not reading values and create table as parameter name.Transformer Pipeline:Raw Data (which actual table in Oracle DB)Snowflake: But Pipeline running, it creates new table as TABLE_NAME, instead of merging on EMP table

1 month ago

lakshmi_narayanan_tDiscovered Fame

asked in Share your Best Practices

Reading XML file in directory it shows error how to fix?

ERROR Shown Is:- ANy One SuGgest to fix ThIs ErRorTimestamp Error Code Error Message Aug 9, 2023, 5:41:08 PM SPOOLDIR_02 SPOOLDIR_02 - Object in file 'beautified.xml' at offset '0' exceeds maximum length (View Stack Trace... ) Aug 9, 2023, 5:41:08 PM SPOOLDIR_01 SPOOLDIR_01 - Failed to process file '/resources/beautified.xml' at position '0': com.streamsets.pipeline.lib.dirspooler.BadSpoolFileException: com.streamsets.pipeline.lib.parser.DataParserException: XML_PARSER_02 - XML object exceeded maximum length: readerId 'beautified.xml', offset '0', maximum length '4096' (View Stack Trace... )

1 month ago

ajinkyaStreamSets Employee

asked in Share your Best Practices

How to tune Oracle CDC parameters to get best performance?

Following parameters can be tuned to get the optimal performance from the Oracle CDC:-Max Batch Size (records) = 20000 Batch Wait Time (secs) = 15 - 30 seconds JDBC Fetch Size for Current Window = 25000 at least JDBC Fetch Size for Past Windows = 25000 at least Time between Session Windows (ms) = 1 ms Time after Session Window Start (ms) = 1ms Parsing Thread Pool Size = 8 (up to 2x the number of CPUs available) NOTE:- Performance depends on the ENV, Infrastructure, Resources, so the values for every environment/use cases may differ. Above mentioned values can be used as a starting point. Tuning performance is trial and error method. Once the optimal values found for the specific environment/use case, those values can be applied.

2 months ago

bapeshirtFan

asked in Share your Best Practices

"Deep Tracks" are not saving on SiriusXM's recent channels.

I almost always listen to Deep Tracks, but for some reason, it never appears under Recent Channels. Does anyone else see this?

2 months ago

itiansgurusFan

asked in Share your Best Practices

Where to Buy Fashion Essentials Hoodie?

We released the new 2023 Essentials hoodie collection from Fog Essentials clothing shop Fear of God Essentials store.

2 months ago

rutuja_pradhanFan

asked in Share your Best Practices

Kerberos setup

Hi,Currently exploring Kerberos authentication with Streamsets. For HDFS.Could you please provide some sample values that need to be set for below properties.kerberos.client.enabled kerberos.client.principal kerberos.client.keytabGetting below error while starting up sdc when kerberos is enabled -Refreshing Kerberos configurationKey for the principal <sdc/hostname> not available in sdc.keytab [Krb5LoginModule] authentication failedUnable to obtain password from user

3 months ago

LavenderStreamSets Employee

posted in Share your Best Practices

How to troubleshoot connection timeout related issues in Azure Synapse stage

Issue: In Azure Synapse stage, there might be some errors that initially look like the issue with loading/writing data to Synapse, but it could be caused by connection timeout:AZURE_STORAGE_07 - Could not get a Stage File Writer instance to write the records: 'AZURE_STORAGE_11 - Could not load file to Azure Storage. AZURE_STORAGE_02 - Azure Synapse load request failed: 'AZURE_DATA_WAREHOUSE_09 - Could not merge staged file com.streamsets.pipeline.api.StageException: AZURE_DATA_WAREHOUSE_00 - Could not perform SQL operationTo determine the root cause, please turn on debug log by adding the following to log4j to further investigate the issue:logger.l5.name = com.streamsets.pipeline.stage.common.synapselogger.l5.level = DEBUGlogger.l6.name = com.streamsets.pipeline.stage.destination.datawarehouselogger.l6.level = TRACEYou might be able to get more hints in debug log and see if it’s related to connection timeout, such as:Connection is not available, request timed out after 30000ms. Channel

3 months ago

LavenderStreamSets Employee

posted in Share your Best Practices

Error HTTP_42 - Failing stage. Connection timeout as per configuration

Issue: You encounter error HTTP_42 - Failing stage. Connection timeout as per configuration in HTTP Client destination.Analysis: This is related to Maximum Request Time stage setting. Maximum Request Time can be viewed as allocated for the entire batch. While the stage is processing responses sequentially, if at any point the Maximum Request Time is exceeded, then that record and any records after it are hit with an HTTP_42. Please increase Maximum Request Time to a higher value to clear out this error.However, from SDC 5.6.0 and below, we discover that there's a bug which makes Maximum Request Time setting conflicting with One request per Batch in HTTP Client Destination. This issue makes Maximum Request Time configuration not picked up when One request per Batch option is enabled. Solution: This bug will be fixed in SDC 5.7.0 and above.Workaround: 1st workaround: Please disable One request per Batch option. 2nd workaround (only applicable to SDC 5.4.0): Please replace these attached

3 months ago

lakshmi_narayanan_tDiscovered Fame

asked in Share your Best Practices

write data into mysql from mongodb how to handle auto increment column

When migrating data from MongoDB to MySQL, I noticed that there is no 'id' column in MongoDB, but there is an 'id' column in MySQL. As a result, I am facing issues when writing data into MySQL. I am looking for assistance in handling the auto-increment column in this scenario. Can anyone provide any guidance or assistance?Additionally, I am exploring the possibility of using the JDBC Tee processor to handle this situation. Could you please advise me on how to use the JDBC Tee processor in this scenario?

4 months ago

LavenderStreamSets Employee

posted in Share your Best Practices

Error java.lang.NoClassDefFoundError: com/sun/activation/registries/LogSupport

Issue: Error java.lang.NoClassDefFoundError:com/sun/activation/registries/LogSupportSolution: This error occurs due Java version is not supported by that SDC version. To check which Java version is supported, you can go to Installation Requirements of that SDC version.For example:For SDC 5.2.0 and below, you can see in Installation Requirements that only Java 8 is supported. For SDC 5.3.0 and above, you can see in Java Versions and Available Features that Java 8, 11, and 17 are supported.You can switch to a different Java version by adding JAVA_HOME variable in $SDC_DIST/libexec/sdc-env.sh or sdcd-env.sh depending on how you start SDC.For example:export JAVA_HOME=/Library/Java/JavaVirtualMachines/openlogic-openjdk-8.jre/Contents/Home

4 months ago

AkshayJadhavStreamSets Employee

posted in Share your Best Practices

JDBC stage is failing with "Unknown system variable 'query_cache_size'" when connecting to MySQL server via Proxy SQL.

Environment:StreamSets Data Collector any version. MySQL database + ProxySQL LB. JDBC producer or JDBC consumer stage.Issue:It has been noticed that when connecting to the MySQL server when ProxySQL Load balancer is placed, the pipeline validation fails with following error:2023-02-22 12:28:12,112 [user:test] [pipeline:test_mysql/mysqlce] [runner:] [thread:preview-pool-16-thread-4] [stage:] ERROR HikariPool - HikariPool-4 - Exception during pool initialization.java.sql.SQLException: Unknown system variable 'query_cache_size' at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129) at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:828) at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:448) at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:241) at com.mysql.cj.jdbc.NonRegisteringDriver.connect

6 months ago

LavenderStreamSets Employee

posted in Share your Best Practices

How to call Runtime Parameters within Functions

Overview:We can call a runtime parameter from within an expression language function by simply entering the parameter name. The semantic of ${PARAMETER} is that PARAMETER is the expression to be evaluated. ${ and } are quotes around the expression to mark where it starts and where it ends. For example, the following expression returns the value of the JDBCConnectionString runtime parameter: ${record:value(JDBCConnectionString)} Use case:We want to create orchestration pipelines which have multiple Job Start stages to run jobs, and we want to specify between DEV and PRD. If we have two jobs for the same pipeline, one with the keyword "DEV" and on with "PRD", we want to use runtime parameter to determine which job the Start Jobs stage runs. Solution:We can set a parameter called ENV and call it either "DEV" or "PRD". Then, in the Start Jobs stage, we use a concat function to try to concatenate the job name with the ENV parameter like this:${str:concat("Job for ", ENV})}

7 months ago

lakshmi_narayanan_tDiscovered Fame

asked in Share your Best Practices

field renamer

when i using field re-namer to perform multipile function with same field it shows error ,if there is any option to perform operation like caps and remove special characters of field with one processor.

7 months ago

Santhosh KumarDiscovered Fame

asked in Share your Best Practices

Change Data Capture

CDC -Hi Team ,i have using Change Data Capture in streamsets it will be working on JDBC Query Consumer to JDBC procedure,its succesfully run only insert process but not update. well how i update .

8 months ago

AkshayJadhavStreamSets Employee

posted in Share your Best Practices

Data collector and Kafka pipeline started failing after Oracle Java upgrade to v1.8u351.

Environment: Streamsets Data Collector All version. Oracle Java 1.8u351.Issue:It has been observed that data collector and Kafka pipeline[with SASL protocol] starts failing after Java upgrade to 1.8u351 version.This error appears on data collector logs:abnormal exit: java.lang.RuntimeException: Could not get Kerberos credentials: javax.security.auth.login.LoginException: Unable to obtain password from userCheck STDERR for more detailsjava.lang.RuntimeException: Could not get Kerberos credentials: javax.security.auth.login.LoginException: Unable to obtain password from user at com.streamsets.datacollector.security.SecurityContext.login(SecurityContext.java:160) at com.streamsets.datacollector.main.Main.doMain(Main.java:94) at com.streamsets.datacollector.main.DataCollectorMain.main(DataCollectorMain.java:41)Kafka pipeline errors:GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: KDC has no support for encryption type (14))]) occurred wh

10 months ago

AkshayJadhavStreamSets Employee

published in Share your Best Practices

STF - Configuration of maximum open file limit is too low: 1024 (expected at least 32768).

Issue:When using the StreamSets Test Framework and trying to run a test it fails with the following error:$ stf test --sdc-version 3.4.0 -s stage/test_dev_stages.py 2019-01-28 09:03:16 AM [INFO] [streamsets.testframework.cli] Pulling Docker image streamsets/testframework:master ... INFO:streamsets.testframework.cli:Pulling Docker image streamsets/testframework:master ... ============================= test session starts ============================== platform linux -- Python 3.6.8, pytest-3.0.4, py-1.7.0, pluggy-0.4.0 benchmark: 3.1.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000) rootdir: /root/tests, inifile: plugins: benchmark-3.1.1 collected 7 itemsstage/test_dev_stages.py 2019-01-28 09:03:27 AM [INFO] [streamsets.testframework.sdc] Starting StreamSets Data Collector 3.4.0 container ... 2019-01-28 09:05:41 AM [ERROR] [streamsets.testframework.sdc] Error reaching Data Col

2852

10 months ago

AkshayJadhavStreamSets Employee

posted in Share your Best Practices

Docker run command is failing with "Container ID 20159 cannot be mapped to a host ID."

Environment:Data Collector Docker Image. Issue: This has been observed that docker run command with any version of data collector image is failing with following error when the user has restrictions from OS side to, not to use higher UIDs.#docker run -d -e datacollector:4.4.1Unable to find image 'datacollector:4.4.1' locally4.4.1: Pulling from datacollector801bfaa63ef2: Pull complete755713f1a3c1: Pull completeeb3cc996a4d2: Pull completecc65d2e08110: Pull completea78edd797c42: Pull complete2837f4f48b17: Extracting [==================================================>] 425.6 MB/425.6 MBfb4a45fc2613: Download completeee56bb53e5ce: Download complete47f24455e547: Download completee0e00f796eb2: Download completeb134ff785401: Download complete/usr/bin/docker-current: failed to register layer: Error processing tar file(exit status 1): Container ID 20159 cannot be mapped to a host ID.See '/usr/bin/docker-current run --help'.In the Data Collector v4.4.1 docker image, we’ve hardcoded the SDC_UI

1 year ago

Page 1 / 2

Badge winners

Sanjeevhas earned the badge Eager to help
AkshayJadhavhas earned the badge Eager to help
john.durkinhas earned the badge Eager to help
samhas earned the badge Eager to help
saleempothiwalahas earned the badge Eager to help

Show all badges

Terms & Conditions

Sign up

Already have an account? Login

Social Login

Username *

E-mail address *

What I do... *

Data Leader Data Architect Data Engineer Data Scientist Other

Company *

Country *

Zip Code *

Marketing Communications

Yes No

Password *

I have read and Agree to the Website Terms of Service and I have read and acknowledged the Privacy Policy.

loginBox.register.email_repeat

Login to the community

No account yet? Create an account

Social Login

Username or Email

Password

Remember me

Forgot password?

Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.

Username or e-mail

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.

Share your Best Practices

Join the Write for Us programBlog

Badge winners

Sign up

Social Login

Login to the community

Social Login

Scanning file for viruses.

This file cannot be downloaded