Share your Best Practices

Share your use cases, tips, and or ideas with other StreamSets members

  • 11 Topics
  • 8 Replies

11 Topics

Akash
Fan
AkashFan
asked in Share your Best Practices

Informatica to Streasets

Hi Team,We have a requirement to transform the informatica mappings into Streamsets. Since there are more than 200 mappings, so it is not possible to create each one of them seprately in Transformer.Can you suggest the best approach and any related link or documentation for this.We are planning to use the Python SDK, but there are lots of challenges in it. 

4
dima
StreamSets Employee
2 months ago
J
Fan
jmanoFan
asked in Share your Best Practices

Data Collector

Hello  new to Streamsets.  my question : In one server can i install 2 DC by creating 2 Env’s  . ? (DevEnvDC and UATEnvDC .) or 1 server 1 DC installation only? thanksJames.  

3
dima
StreamSets Employee
2 months ago
mblahay
Roadie
mblahayRoadie
posted in Share your Best Practices

Splitting a fixed width array

I’ve been working with some fixed width records recently and one of the tasks was to split a string field into an array of like sized strings. The problem however seemed insurmountable using the field splitter stage as a delimiter is required. I ended up using the Jython evaluator to accomplish the task. Later on, I happened across an article talking about a similar problem in the context of using the Java regular expression engine. The article provided a solution using the \G boundary match combined with a look behind.For example:  (?<=\G.{4})When used in the field splitter stage this regex will split the string into equal length parts that are 4 characters in size.  The regex essentially creates a zero length delimiter which enforces a 4 character separation between the last delimiter and the current one.Shame on me for loosing the url for the original article. If I come across it again I will be sure to site it here.

1
Drew Kreiger
StreamSets Employee
4 months ago
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Share your Best Practices

WARN "Unable to get Instrumentation. Dynamic Attach failed" while booting SDC service.

Issue:The SDC is showing following error while start the process on sone of the java version however this will not stop the data collector process from booting. This can be seen while booting only.# WARNING: Unable to get Instrumentation. Dynamic Attach failed. You may add this JAR as -javaagent manually, or supply -Djdk.attach.allowAttachSelf# WARNING: Unable to attach Serviceability Agent. Unable to attach even with module exceptions: [org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.openjdk.jol.vm.sa.SASupportException: Sense failed.] Solution:There are 2 ways to suppress the warning message:We can added the global variable in linux which will enable this property for all the java process running on linux machine, however the JAVA_TOOL_OPTIONS variable will be unset after the reboot. If you want to enable it, for SDC process only then follow the second solution. export JAVA_TOOL_OPTIONS="-Djdk.attach.allowAttachSel

0
AkshayJadhav
StreamSets Employee
29 days ago
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Share your Best Practices

Unable to activate Control Hub with “ERROR: java.io.IOException: java.security.SignatureException: Signature length not correct: got 512 but was expecting 256”

Question:When trying to apply the activation key for Control Hub On-Pre from the CLI, one might receive an error:ERROR: java.io.IOException: java.security.SignatureException: Signature length not correct: got 512 but was expecting 256 Answer:Control Hub is unable to read or reach the streamset-activation.jar file in the lib directory. Under the $DPM_HOME/apps/security/lib, check to see if there is both a 'streamsets-activation-<version>.jar' and a 'streamsets-activation-dev-<version>.jar'.  If there is a 'streamsets-activation-dev-<version>.jar', remove the dev jar, as it is meant only for the dev builds.  If there is no jar in the directory, a user may have deleted the file or was absent from the installation process.  To retrieve a new jar file, extract the jar file from a tarball of the Control Hub On-Prem version that one is using.

0
AkshayJadhav
StreamSets Employee
13 days ago
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
published in Share your Best Practices

STF - Configuration of maximum open file limit is too low: 1024 (expected at least 32768).

Issue:When using the StreamSets Test Framework and trying to run a test it fails with the following error:$ stf test --sdc-version 3.4.0 -s stage/test_dev_stages.py 2019-01-28 09:03:16 AM [INFO] [streamsets.testframework.cli] Pulling Docker image streamsets/testframework:master ... INFO:streamsets.testframework.cli:Pulling Docker image streamsets/testframework:master ... ============================= test session starts ============================== platform linux -- Python 3.6.8, pytest-3.0.4, py-1.7.0, pluggy-0.4.0 benchmark: 3.1.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000) rootdir: /root/tests, inifile: plugins: benchmark-3.1.1 collected 7 itemsstage/test_dev_stages.py 2019-01-28 09:03:27 AM [INFO] [streamsets.testframework.sdc] Starting StreamSets Data Collector 3.4.0 container ... 2019-01-28 09:05:41 AM [ERROR] [streamsets.testframework.sdc] Error reaching Data Col

60
AkshayJadhav
StreamSets Employee
13 days ago
Brenna
StreamSets Employee
BrennaStreamSets Employee
posted in Share your Best Practices

Docker Warning Experienced with a M1 Chip Macbook

If you see the below warning when creating an engine with Docker on a macbook with an M1 chip,WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requestedadd --platform linux/amd64 to your docker run command.

0
Brenna
StreamSets Employee
4 months ago
Dash
Headliner
DashSenior Technical Evangelist and Developer Advocate at Snowflake
posted in Share your Best Practices

Oracle CDC To Snowflake

If you’re looking for a step-by-step walkthrough of migrating data from Oracle to Snowflake via Change Data Capture on StreamSets DataOps Platform, refer to this guide. Cheers,Dash

0
Dash
Headliner
9 months ago
Bikram
Opening Band
BikramOpening Band
asked in Share your Best Practices

How to remove special characters from field names in streamsets data collector?

Solution.The issue can be handled by Field mapper processor. Below the syntax for the same. ${str:replaceAll(f:name(), '[.\\-/()!&%^$#@]','')} Attached the screen shot for the reference. NB : The issue also can be handled in different ways e;g(Expression Evaluator, scripting processor etc) 

0
Bikram
Opening Band
2 months ago
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Share your Best Practices

How to connect to Azure dedicated SQL pool via Azure Synapse SQL?

In this article, we'll configure the Azure Synapse Dedicated SQL pool and insert some records from Mysql.Platform:Streamsets Data Collector version 5.0 MySql version 8.0.28.Pre-requisite:Mysql connector driver for JDBC query consumer stage. Install Azure Synapse Enterprise Library 1.1.0 from PackageManager to enableAzure Synapse SQL destination. Configuration:Step1: When you create the workspace in Azure Synapse, you will be creating a storage account and workspace however it does not create a SQL pool. You would need to create the SQL pool from:Azure Analytics » Click on Workspace web URL » Manage and then click on “new”. This will create the Azure SQL pool.Once done, you will be able to see the SQL pool in the overview tab. “akshaytest” is the new sql pool here. Step2: Develop the pipeline with JDBC Query Consumer origin and Azure Synapse SQL destination.In the JDBC consumer, we would be reading all the data from the “Azure” mysql database which is hosted on the local machine.These a

0
AkshayJadhav
StreamSets Employee
1 month ago
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Share your Best Practices

Why new line is getting added in Kafka when using Delimited "RFC4180_CSV" as data format?

Scenario:A new line is getting added in the every records when using Delimited RFC4180_CSV in Kafka Producer. However, if we use any other data format we would not see the extra line issue. Here are the screenshots of configuration and console-consumer script.This the DataFormat configuration.Pipeline has processed 10 messages.Kafka-console-consumer output.In the above output, I could see an extra line is added in every records. We tested the pipeline with Replace New Line Characters configuration and found that it does not add extra line in every records however, it adds the line after every batch of records i.e. a new line after 1000 records. Solution:The new line is introduced by the CSV format that we are using in pipeline means kafka producer is working as expected.[1]. Common Format and MIME Type for Comma-Separated Values (CSV) Files - Page2-3https://datatracker.ietf.org/doc/html/rfc4180++++++++++++CSV: It is an example of a "flat file" format. It is a delimited data format that

0
AkshayJadhav
StreamSets Employee
1 month ago

Badge winners

Show all badges
Terms & ConditionsCookie settings