Skip to main content

Share your Best Practices

Share your use cases, tips, and or ideas with other StreamSets members

  • 59 Topics
  • 25 Replies
59 Topics
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Share your Best Practices
Unable to activate Control Hub with “ERROR: java.io.IOException: java.security.SignatureException: Signature length not correct: got 512 but was expecting 256”

Question:When trying to apply the activation key for Control Hub On-Pre from the CLI, one might receive an error:ERROR: java.io.IOException: java.security.SignatureException: Signature length not correct: got 512 but was expecting 256 Answer:Control Hub is unable to read or reach the streamset-activation.jar file in the lib directory. Under the $DPM_HOME/apps/security/lib, check to see if there is both a 'streamsets-activation-<version>.jar' and a 'streamsets-activation-dev-<version>.jar'.  If there is a 'streamsets-activation-dev-<version>.jar', remove the dev jar, as it is meant only for the dev builds.  If there is no jar in the directory, a user may have deleted the file or was absent from the installation process.  To retrieve a new jar file, extract the jar file from a tarball of the Control Hub On-Prem version that one is using.

1560
AkshayJadhav
StreamSets Employee
2 years ago
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Share your Best Practices
WARN "Unable to get Instrumentation. Dynamic Attach failed" while booting SDC service.

Issue:The SDC is showing following error while start the process on sone of the java version however this will not stop the data collector process from booting. This can be seen while booting only.# WARNING: Unable to get Instrumentation. Dynamic Attach failed. You may add this JAR as -javaagent manually, or supply -Djdk.attach.allowAttachSelf# WARNING: Unable to attach Serviceability Agent. Unable to attach even with module exceptions: [org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.openjdk.jol.vm.sa.SASupportException: Sense failed.] Solution:There are 2 ways to suppress the warning message:We can added the global variable in linux which will enable this property for all the java process running on linux machine, however the JAVA_TOOL_OPTIONS variable will be unset after the reboot. If you want to enable it, for SDC process only then follow the second solution. export JAVA_TOOL_OPTIONS="-Djdk.attach.allowAttachSel

15080
AkshayJadhav
StreamSets Employee
2 years ago
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Share your Best Practices
Why new line is getting added in Kafka when using Delimited "RFC4180_CSV" as data format?

Scenario:A new line is getting added in the every records when using Delimited RFC4180_CSV in Kafka Producer. However, if we use any other data format we would not see the extra line issue. Here are the screenshots of configuration and console-consumer script.This the DataFormat configuration.Pipeline has processed 10 messages.Kafka-console-consumer output.In the above output, I could see an extra line is added in every records. We tested the pipeline with Replace New Line Characters configuration and found that it does not add extra line in every records however, it adds the line after every batch of records i.e. a new line after 1000 records. Solution:The new line is introduced by the CSV format that we are using in pipeline means kafka producer is working as expected.[1]. Common Format and MIME Type for Comma-Separated Values (CSV) Files - Page2-3https://datatracker.ietf.org/doc/html/rfc4180++++++++++++CSV: It is an example of a "flat file" format. It is a delimited data format that

5050
AkshayJadhav
StreamSets Employee
2 years ago
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Share your Best Practices
How to connect to Azure dedicated SQL pool via Azure Synapse SQL?

In this article, we'll configure the Azure Synapse Dedicated SQL pool and insert some records from Mysql.Platform:Streamsets Data Collector version 5.0 MySql version 8.0.28.Pre-requisite:Mysql connector driver for JDBC query consumer stage. Install Azure Synapse Enterprise Library 1.1.0 from PackageManager to enableAzure Synapse SQL destination. Configuration:Step1: When you create the workspace in Azure Synapse, you will be creating a storage account and workspace however it does not create a SQL pool. You would need to create the SQL pool from:Azure Analytics » Click on Workspace web URL » Manage and then click on “new”. This will create the Azure SQL pool.Once done, you will be able to see the SQL pool in the overview tab. “akshaytest” is the new sql pool here. Step2: Develop the pipeline with JDBC Query Consumer origin and Azure Synapse SQL destination.In the JDBC consumer, we would be reading all the data from the “Azure” mysql database which is hosted on the local machine.These a

1690
AkshayJadhav
StreamSets Employee
2 years ago
J
Fan
jmanoFan
asked in Share your Best Practices
Data Collector

Hello  new to Streamsets.  my question : In one server can i install 2 DC by creating 2 Env’s  . ? (DevEnvDC and UATEnvDC .) or 1 server 1 DC installation only? thanksJames.  

553
dima
StreamSets Employee
2 years ago
Akash
Fan
AkashFan
asked in Share your Best Practices
Informatica to Streasets

Hi Team,We have a requirement to transform the informatica mappings into Streamsets. Since there are more than 200 mappings, so it is not possible to create each one of them seprately in Transformer.Can you suggest the best approach and any related link or documentation for this.We are planning to use the Python SDK, but there are lots of challenges in it. 

1234
dima
StreamSets Employee
2 years ago
A
Anonymous
posted in Share your Best Practices
Docker Warning Experienced with a M1 Chip Macbook

If you see the below warning when creating an engine with Docker on a macbook with an M1 chip,WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requestedadd --platform linux/amd64 to your docker run command.

1270
A
2 years ago
mblahay
Discovered Fame
mblahayDiscovered Fame
posted in Share your Best Practices
Splitting a fixed width array

I’ve been working with some fixed width records recently and one of the tasks was to split a string field into an array of like sized strings. The problem however seemed insurmountable using the field splitter stage as a delimiter is required. I ended up using the Jython evaluator to accomplish the task. Later on, I happened across an article talking about a similar problem in the context of using the Java regular expression engine. The article provided a solution using the \G boundary match combined with a look behind.For example:  (?<=\G.{4})When used in the field splitter stage this regex will split the string into equal length parts that are 4 characters in size.  The regex essentially creates a zero length delimiter which enforces a 4 character separation between the last delimiter and the current one.Shame on me for loosing the url for the original article. If I come across it again I will be sure to site it here.

1431
Drew Kreiger
Rock star
2 years ago
Dash
Headliner
DashSenior Technical Evangelist and Developer Advocate at Snowflake
posted in Share your Best Practices
Oracle CDC To Snowflake

If you’re looking for a step-by-step walkthrough of migrating data from Oracle to Snowflake via Change Data Capture on StreamSets DataOps Platform, refer to this guide. Cheers,Dash

700
Dash
Headliner
3 years ago

Badge winners

Show all badges
Terms & Conditions