- - Knowledge base
Product Updates
Groups
Join Build It With Brenna

Build it with Brenna: Transformer for Snowflake

(Wed, 31 May, 17:00)

Home
Community overview
Platform Q&A
Share your Best Practices

Share your Best Practices

Share your use cases, tips, and or ideas with other StreamSets members

24 Topics
16 Replies

When you subscribe we will email you when there is a new topic in this category

24 Topics

Recently active

Most replies Most views Newest first

LavenderStreamSets Employee

posted in Share your Best Practices

How to troubleshoot connection timeout related issues in Azure Synapse stage

Issue: In Azure Synapse stage, there might be some errors that initially look like the issue with loading/writing data to Synapse, but it could be caused by connection timeout:AZURE_STORAGE_07 - Could not get a Stage File Writer instance to write the records: 'AZURE_STORAGE_11 - Could not load file to Azure Storage. AZURE_STORAGE_02 - Azure Synapse load request failed: 'AZURE_DATA_WAREHOUSE_09 - Could not merge staged file com.streamsets.pipeline.api.StageException: AZURE_DATA_WAREHOUSE_00 - Could not perform SQL operationTo determine the root cause, please turn on debug log by adding the following to log4j to further investigate the issue:logger.l5.name = com.streamsets.pipeline.stage.common.synapselogger.l5.level = DEBUGlogger.l6.name = com.streamsets.pipeline.stage.destination.datawarehouselogger.l6.level = TRACEYou might be able to get more hints in debug log and see if it’s related to connection timeout, such as:Connection is not available, request timed out after 30000ms. Channel

3 days ago

LavenderStreamSets Employee

posted in Share your Best Practices

Error HTTP_42 - Failing stage. Connection timeout as per configuration

Issue: Error HTTP_42 - Failing stage. Connection timeout as per configuration in HTTP Client processor/destination.Solution: We recommend to increase the "Max Request Timeout" configuration in the HTTP stage. (Default is 60 seconds).If increasing "Max Request Timeout" doesn’t solve the error, please perform networking testings such as generating tcpdump to investigate your networking environment and open a ticket with Support for further troubleshooting.

8 days ago

lakshmi_narayanan_tDiscovered Fame

asked in Share your Best Practices

write data into mysql from mongodb how to handle auto increment column

When migrating data from MongoDB to MySQL, I noticed that there is no 'id' column in MongoDB, but there is an 'id' column in MySQL. As a result, I am facing issues when writing data into MySQL. I am looking for assistance in handling the auto-increment column in this scenario. Can anyone provide any guidance or assistance?Additionally, I am exploring the possibility of using the JDBC Tee processor to handle this situation. Could you please advise me on how to use the JDBC Tee processor in this scenario?

27 days ago

LavenderStreamSets Employee

posted in Share your Best Practices

Error java.lang.NoClassDefFoundError: com/sun/activation/registries/LogSupport

Issue: Error java.lang.NoClassDefFoundError:com/sun/activation/registries/LogSupportSolution: This error occurs due Java version is not supported by that SDC version. To check which Java version is supported, you can go to Installation Requirements of that SDC version.For example:For SDC 5.2.0 and below, you can see in Installation Requirements that only Java 8 is supported. For SDC 5.3.0 and 5.4.0, you can see in Java Versions and Available Features that Java 8, 11, and 17 are supported.You can switch to a different Java version by adding JAVA_HOME variable in $SDC_DIST/libexec/sdc-env.sh.export JAVA_HOME=/Library/Java/JavaVirtualMachines/openlogic-openjdk-8.jre/Contents/Home

1 month ago

AkshayJadhavStreamSets Employee

posted in Share your Best Practices

JDBC stage is failing with "Unknown system variable 'query_cache_size'" when connecting to MySQL server via Proxy SQL.

Environment:StreamSets Data Collector any version. MySQL database + ProxySQL LB. JDBC producer or JDBC consumer stage.Issue:It has been noticed that when connecting to the MySQL server when ProxySQL Load balancer is placed, the pipeline validation fails with following error:2023-02-22 12:28:12,112 [user:test] [pipeline:test_mysql/mysqlce] [runner:] [thread:preview-pool-16-thread-4] [stage:] ERROR HikariPool - HikariPool-4 - Exception during pool initialization.java.sql.SQLException: Unknown system variable 'query_cache_size' at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129) at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:828) at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:448) at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:241) at com.mysql.cj.jdbc.NonRegisteringDriver.connect

2 months ago

LavenderStreamSets Employee

posted in Share your Best Practices

How to call Runtime Parameters within Functions

Overview:We can call a runtime parameter from within an expression language function by simply entering the parameter name. The semantic of ${PARAMETER} is that PARAMETER is the expression to be evaluated. ${ and } are quotes around the expression to mark where it starts and where it ends. For example, the following expression returns the value of the JDBCConnectionString runtime parameter: ${record:value(JDBCConnectionString)} Use case:We want to create orchestration pipelines which have multiple Job Start stages to run jobs, and we want to specify between DEV and PRD. If we have two jobs for the same pipeline, one with the keyword "DEV" and on with "PRD", we want to use runtime parameter to determine which job the Start Jobs stage runs. Solution:We can set a parameter called ENV and call it either "DEV" or "PRD". Then, in the Start Jobs stage, we use a concat function to try to concatenate the job name with the ENV parameter like this:${str:concat("Job for ", ENV})}

3 months ago

lakshmi_narayanan_tDiscovered Fame

asked in Share your Best Practices

field renamer

when i using field re-namer to perform multipile function with same field it shows error ,if there is any option to perform operation like caps and remove special characters of field with one processor.

4 months ago

Santhosh KumarDiscovered Fame

asked in Share your Best Practices

Change Data Capture

CDC -Hi Team ,i have using Change Data Capture in streamsets it will be working on JDBC Query Consumer to JDBC procedure,its succesfully run only insert process but not update. well how i update .

5 months ago

AkshayJadhavStreamSets Employee

posted in Share your Best Practices

Data collector and Kafka pipeline started failing after Oracle Java upgrade to v1.8u351.

Environment: Streamsets Data Collector All version. Oracle Java 1.8u351.Issue:It has been observed that data collector and Kafka pipeline[with SASL protocol] starts failing after Java upgrade to 1.8u351 version.This error appears on data collector logs:abnormal exit: java.lang.RuntimeException: Could not get Kerberos credentials: javax.security.auth.login.LoginException: Unable to obtain password from userCheck STDERR for more detailsjava.lang.RuntimeException: Could not get Kerberos credentials: javax.security.auth.login.LoginException: Unable to obtain password from user at com.streamsets.datacollector.security.SecurityContext.login(SecurityContext.java:160) at com.streamsets.datacollector.main.Main.doMain(Main.java:94) at com.streamsets.datacollector.main.DataCollectorMain.main(DataCollectorMain.java:41)Kafka pipeline errors:GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: KDC has no support for encryption type (14))]) occurred wh

6 months ago

AkshayJadhavStreamSets Employee

published in Share your Best Practices

STF - Configuration of maximum open file limit is too low: 1024 (expected at least 32768).

Issue:When using the StreamSets Test Framework and trying to run a test it fails with the following error:$ stf test --sdc-version 3.4.0 -s stage/test_dev_stages.py 2019-01-28 09:03:16 AM [INFO] [streamsets.testframework.cli] Pulling Docker image streamsets/testframework:master ... INFO:streamsets.testframework.cli:Pulling Docker image streamsets/testframework:master ... ============================= test session starts ============================== platform linux -- Python 3.6.8, pytest-3.0.4, py-1.7.0, pluggy-0.4.0 benchmark: 3.1.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000) rootdir: /root/tests, inifile: plugins: benchmark-3.1.1 collected 7 itemsstage/test_dev_stages.py 2019-01-28 09:03:27 AM [INFO] [streamsets.testframework.sdc] Starting StreamSets Data Collector 3.4.0 container ... 2019-01-28 09:05:41 AM [ERROR] [streamsets.testframework.sdc] Error reaching Data Col

2122

6 months ago

AkshayJadhavStreamSets Employee

posted in Share your Best Practices

Docker run command is failing with "Container ID 20159 cannot be mapped to a host ID."

Environment:Data Collector Docker Image. Issue: This has been observed that docker run command with any version of data collector image is failing with following error when the user has restrictions from OS side to, not to use higher UIDs.#docker run -d -e datacollector:4.4.1Unable to find image 'datacollector:4.4.1' locally4.4.1: Pulling from datacollector801bfaa63ef2: Pull complete755713f1a3c1: Pull completeeb3cc996a4d2: Pull completecc65d2e08110: Pull completea78edd797c42: Pull complete2837f4f48b17: Extracting [==================================================>] 425.6 MB/425.6 MBfb4a45fc2613: Download completeee56bb53e5ce: Download complete47f24455e547: Download completee0e00f796eb2: Download completeb134ff785401: Download complete/usr/bin/docker-current: failed to register layer: Error processing tar file(exit status 1): Container ID 20159 cannot be mapped to a host ID.See '/usr/bin/docker-current run --help'.In the Data Collector v4.4.1 docker image, we’ve hardcoded the SDC_UI

8 months ago

AkshayJadhavStreamSets Employee

posted in Share your Best Practices

Snowflake File Uploader is failing with "Authentication token has expired, The user must authenticate again" error when pipeline is idle for long time.

Environment:Streamsets Data Collector. Streamsets Library 1.X Issue: This has been observed that snowflake pipeline is failing with “user authentication” error when pipeline is idle for more than 4-6+ hours. This is the error stack:com.streamsets.pipeline.api.base.OnRecordErrorException: SNOWFLAKE_65 - File upload to Snowflake failed: net.snowflake.client.jdbc.SnowflakeReauthenticationRequest: Authentication token has expired. The user must authenticate again. at com.streamsets.pipeline.stage.destination.snowflake.uploader.SnowflakeFileUploaderTarget.write(SnowflakeFileUploaderTarget.java:155) at com.streamsets.pipeline.api.base.configurablestage.DTarget.write(DTarget.java:34) at com.streamsets.datacollector.runner.StageRuntime.lambda$execute$2(StageRuntime.java:291) at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:232) at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:299) at com.streamsets.datacollector.runne

9 months ago

rekakomaFan

posted in Share your Best Practices

Data Engineer vs Data Science

In Data Engineer vs Data Science, Data is both - the currency of the century as well as the most in-demand commodity. This increase in the value of data has also led to an unprecedented rise in the demand for talent who can make the most effective use of this data. In fact, roles such as Data Scientist and Data Engineer, are among the most desirable employment roles today, with highly scalable growth opportunities. Data engineering is the process of developing and constructing large-scale data collection, storage, and analysis systems. It's a wide-ranging discipline with applications in almost every industry. Data science is a discipline that blends subject matter expertise, computer skills, and mathematical and statistical knowledge to extract useful insights from data.

9 months ago

rekakomaFan

posted in Share your Best Practices

What is data science?

Data science is the science of sorting, processing, and extracting data from multiple sources and applying statistics, algorithms, and analysis to transform the data into meaningful insights.Since its evolution in the 1960s, data science has helped businessmen realize the value of data in growing businesses by tapping into the voluminous data collecting dust in the servers. Data science has helped file the task of sorting, structuring, analyzing, and suggesting improvisations in business tactics and strategies all under one head - Data Scientist. A Data Scientist peruses this data, cultivates business acumen, and offers insights and constructive criticism on the performance of a business or a company while simultaneously making use of the abundant data available for further advancements like Machine Learning, AI (Artificial Intelligence), and cybersecurity.

9 months ago

AkshayJadhavStreamSets Employee

asked in Share your Best Practices

JDBC Driver showing warning when the data source is MySQL.

Environment: StreamSets Data Collector 4.X. JDBC Producer stage with Mysql.Issue:This has been noticed that data collector is showing following warning when pipeline is using MySQL as a data source.You can find the warning on the console where you start the data collector or in the process logs /tmp/engine-PID.out file.Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.We have tested this internally on Data Collector 4.X series with MySQL version 8.28.0 and mysql-connector-java-8.0.28.jar file.Solution:The warning message is coming from the driver itself. It's completely fine and there is no need to worry but we have to keep it like that for backward compatibility.The way the DriverManager works is similar to a class loader, you load drivers, but you don't know which one will be picked up. So the new driver uses

9 months ago

AkshayJadhavStreamSets Employee

posted in Share your Best Practices

Unable to activate Control Hub with “ERROR: java.io.IOException: java.security.SignatureException: Signature length not correct: got 512 but was expecting 256”

Question:When trying to apply the activation key for Control Hub On-Pre from the CLI, one might receive an error:ERROR: java.io.IOException: java.security.SignatureException: Signature length not correct: got 512 but was expecting 256 Answer:Control Hub is unable to read or reach the streamset-activation.jar file in the lib directory. Under the $DPM_HOME/apps/security/lib, check to see if there is both a 'streamsets-activation-<version>.jar' and a 'streamsets-activation-dev-<version>.jar'. If there is a 'streamsets-activation-dev-<version>.jar', remove the dev jar, as it is meant only for the dev builds. If there is no jar in the directory, a user may have deleted the file or was absent from the installation process. To retrieve a new jar file, extract the jar file from a tarball of the Control Hub On-Prem version that one is using.

11 months ago

AkshayJadhavStreamSets Employee

posted in Share your Best Practices

WARN "Unable to get Instrumentation. Dynamic Attach failed" while booting SDC service.

Issue:The SDC is showing following error while start the process on sone of the java version however this will not stop the data collector process from booting. This can be seen while booting only.# WARNING: Unable to get Instrumentation. Dynamic Attach failed. You may add this JAR as -javaagent manually, or supply -Djdk.attach.allowAttachSelf# WARNING: Unable to attach Serviceability Agent. Unable to attach even with module exceptions: [org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.openjdk.jol.vm.sa.SASupportException: Sense failed.] Solution:There are 2 ways to suppress the warning message:We can added the global variable in linux which will enable this property for all the java process running on linux machine, however the JAVA_TOOL_OPTIONS variable will be unset after the reboot. If you want to enable it, for SDC process only then follow the second solution. export JAVA_TOOL_OPTIONS="-Djdk.attach.allowAttachSel

1 year ago

AkshayJadhavStreamSets Employee

posted in Share your Best Practices

Why new line is getting added in Kafka when using Delimited "RFC4180_CSV" as data format?

Scenario:A new line is getting added in the every records when using Delimited RFC4180_CSV in Kafka Producer. However, if we use any other data format we would not see the extra line issue. Here are the screenshots of configuration and console-consumer script.This the DataFormat configuration.Pipeline has processed 10 messages.Kafka-console-consumer output.In the above output, I could see an extra line is added in every records. We tested the pipeline with Replace New Line Characters configuration and found that it does not add extra line in every records however, it adds the line after every batch of records i.e. a new line after 1000 records. Solution:The new line is introduced by the CSV format that we are using in pipeline means kafka producer is working as expected.[1]. Common Format and MIME Type for Comma-Separated Values (CSV) Files - Page2-3https://datatracker.ietf.org/doc/html/rfc4180++++++++++++CSV: It is an example of a "flat file" format. It is a delimited data format that

1 year ago

AkshayJadhavStreamSets Employee

posted in Share your Best Practices

BrennaStreamSets Employee

posted in Share your Best Practices

Docker Warning Experienced with a M1 Chip Macbook

If you see the below warning when creating an engine with Docker on a macbook with an M1 chip,WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requestedadd --platform linux/amd64 to your docker run command.

1 year ago

mblahayDiscovered Fame

posted in Share your Best Practices

Splitting a fixed width array

I’ve been working with some fixed width records recently and one of the tasks was to split a string field into an array of like sized strings. The problem however seemed insurmountable using the field splitter stage as a delimiter is required. I ended up using the Jython evaluator to accomplish the task. Later on, I happened across an article talking about a similar problem in the context of using the Java regular expression engine. The article provided a solution using the \G boundary match combined with a look behind.For example: (?<=\G.{4})When used in the field splitter stage this regex will split the string into equal length parts that are 4 characters in size. The regex essentially creates a zero length delimiter which enforces a 4 character separation between the last delimiter and the current one.Shame on me for loosing the url for the original article. If I come across it again I will be sure to site it here.

1 year ago

DashSenior Technical Evangelist and Developer Advocate at Snowflake

posted in Share your Best Practices

Oracle CDC To Snowflake

If you’re looking for a step-by-step walkthrough of migrating data from Oracle to Snowflake via Change Data Capture on StreamSets DataOps Platform, refer to this guide. Cheers,Dash

1 year ago

Badge winners

AkshayJadhavhas earned the badge Eager to help
john.durkinhas earned the badge Eager to help
samhas earned the badge Eager to help
saleempothiwalahas earned the badge Eager to help
Bikramhas earned the badge Community Champion 🏅

Show all badges

Terms & Conditions

Sign up

Already have an account? Login

Social Login

Username *

E-mail address *

What I do... *

Data Leader Data Architect Data Engineer Data Scientist Other

Company *

Country *

Zip Code *

Marketing Communications

Yes No

Password *

I have read and Agree to the Website Terms of Service and I have read and acknowledged the Privacy Policy.

loginBox.register.email_repeat

Login to the community

No account yet? Create an account

Social Login

Username or Email

Password

Remember me

Forgot password?

Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.

Username or e-mail

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.