Community Articles

Contribute and publish your articles/blogs here showcasing you usage, architecture, and know how of the StreamSets DataOps Platform. THIS IS NOT THE Q&A CATEGORY. Head over to the "Got a Question?" Category to ask your question.

  • 83 Topics
  • 114 Replies

83 Topics

Mike Calma
StreamSets Employee
Mike CalmaStreamSets Employee
posted in Community Articles

JDBC Lookup Stage Performance Troubleshooting

How to troubleshoot: JDBC Lookup Performance Issues When facing performance issues within the JDBC Lookup stage there are a variety of different variables which may cause or contribute to the issue. Provided below are some initial techniques which you can add to your tool belt to assist you in diagnosing this issue.* To note, while this article is targeted specifically for diagnosing JDBC Lookup issues the same principles of steps 1 & 2 can be used towards JDBC Query Consumers.Step 1: Enabling DEBUG LoggingEnabling DEBUG can often provide a lot of the information you need to determine a next action for investigation. It can provide additional context to error messages which you may have observed previously, but it may also show errors which were previously not available. Additionally, it can also provide information which - while not an obvious error - can provide crucial insight into the pipeline operation such as statistical data. Debug can be configured in Data Collector via the

0
Mike Calma
StreamSets Employee
3 days ago
Rishi
StreamSets Employee
RishiStreamSets Employee
posted in Community Articles

How to Modify Fragment icon ?

By default, the pipeline canvas displays each fragment in the pipeline as a single stage with a puzzle piece icon: Now you can modify the icon to represent the fragment processing logic. You can select from a set of predefined icons, or you can upload an image to use as the icon. Uploaded images cannot exceed 100 KB in size.While viewing a fragment in edit mode, click the General tab in the properties panel. Click the Edit icon next to the Icon property: In the Icons dialog box, select one of the predefined icons or upload a custom image to use as the icon. Note: You can upload a single image to the dialog box. Uploading another image replaces the existing uploaded image. After you publish the fragment, all pipelines that use that fragment version display the fragment as a single stage with the modified icon. More information about fragments could be found here. 

0
Rishi
StreamSets Employee
9 days ago
Pradeep
StreamSets Employee
PradeepStreamSets Employee
posted in Community Articles

Understanding and Resolving the NoClassDefFoundError: Common Causes and Solutions

The NoClassDefFoundError is a runtime error in Java that occurs if the Java Virtual Machine (JVM) or a ClassLoader instance attempts to load the definition of a class that could not be found. The class definition exists at compile-time but is not available at runtime.The definition of the class is attempted to be loaded as part of a normal method call or creating an instance of the class using the new expression and no definition of the class could be found. Therefore, it can occur during the linking or loading of the unavailable class.Common causes of the class definition being unavailable at runtime and solutions to try:Missing JAR file: First, identify the JAR file that contains the class for which you are receiving the NoClassDefFoundError. Once identified, verify if the JAR file exists in the installation directory structure of the StreamSets engine. Alternatively, check if it is located in other paths referenced by the data plane engine (e.g., the USER_LIBRARIES_DIR variable, whi

0
Pradeep
StreamSets Employee
9 days ago
xagavin
Fan
xagavinFan
asked in Community Articles

​Use Oracle CDC Client the record value warped UNISTR function

​I used StreamSets Data Collector 5.4.0, Oracle character set is AL32UTF8, In preview mode, I see that the record value is wrapped in a layer of UNISTR functions, convert to unicodelike this :Oracle CDC Client config :is it something wrong with my JDBC config or how to set the character set ? thanks

6
xagavin
Fan
11 days ago
pedro
StreamSets Employee
pedroStreamSets Employee
posted in Community Articles

How to use Content Type ‘multipart/form-data’ in the HTTP Client stage

SDC’s HTTP Client stage does not have a native option to send HTTP requests using the Content Type ‘multipart/form-data’. Using the default options will result in SDC making the call seemingly correctly but it will not add the required boundary characters that a request using this kind of Content Type requires. However, we can force SDC to send the call in the correct multipart/form-data’ format. To do this you will need to set these parameters in the following way:*** For this example, the request SDC would make is equivalent to the following curl command:curl --location 'https://test.url.com/api/jobs' \--header 'Accept: application/json' \--header 'Authorization: Bearer *********************' \--header 'Cookie: staging-connections-canary="45412ba154f512d34"' \--form 'options="{\"options\":{},\"url\":\"https://test.url.com\",\"sslVersion\":\"TLS 1.2\",\"password\":xxxxxxxxxx,\"userId\":xxxxxxxxxx,\"displayName\":\"Pedro\",\"jobName\":\"pedro_test_job\",\"batchsize\":1000,\"waitTimeout

0
pedro
StreamSets Employee
12 days ago
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Community Articles

How to load the data to the elasticsearch destination?

In this article, our focus will be on utilizing the Elasticsearch 7.9.0 library to load data from a Kafka topic into Elasticsearch.We are going to use following environment:Kafka Consumer Origin [Kafka 2.7] Elasticsearch Destination [Elasticsearch Library 7.9.0]Currently, the source Kafka topic "elastic" contains approximately 5327 records. Now, let's delve into the details of our pipeline setup:Stage1: Consumer will fetch records from topic with the rate of 1000 records per batch. Stage2: We are going to remove the some unnecessary field from each record. Stage3: Credit card details will be masked and only last 4 digit will be visible.  Stage4: Upload the data to Elastic with index customer_data.Data Collector Pipeline flow.To run this pipeline, We need to setup the Elasticsearch deployment and get the username password + endpoint details from the ElasticSearch » Deployment section.Elasticsearch Deployment details. After successfully validating the pipeline, we proceeded to start the

0
AkshayJadhav
StreamSets Employee
15 days ago
molo
Fan
moloFan
asked in Community Articles

Core SDC Tarball initialization failed

Local deployment of, error reported during startup

1
AkshayJadhav
StreamSets Employee
16 days ago
btsmerchstore45
Fan
btsmerchstore45Fan
asked in Community Articles

Is it possible to configure the transformer pipeline to run on an AWS EMR cluster?

I have created a simple transformer pipeline to run on an EMR cluster. In Cluster configuration, I have used the necessary configuration details.I am facing the following issue:

1
kanyewestshirt89
18 days ago
blakeorsak
Fan
blakeorsakFan
posted in Community Articles

JDBC Producer with Multi-Row Operations Parse Error (JDBC_14)

Issue:  When using a JDBC Producer destination with Multi-Row Operation enabled, the following error is thrown:  JDBC_14 - Error processing batch: [SQLState:S0001][ErrorCode:103010] Parse error at line: 1, column: 58: Incorrect syntax near ','. SQLState: S0001 Error Code: 103010 Message: Parse error at line: 1, column: 58: Incorrect syntax near ','. Solution:  This error is the result of certain databases not supporting the underlying SQL statements made by the JDBC Producer when this setting is enabled.  For example, for Multi-Row inserts, the JDBC Producer uses the following SQL statement: INSERT INTO <table name> (<col1>, <col2>, <col3>)      VALUES (<record1 field1>,<record1 field2>,<record1 field3>),      (<r2 f1>,<r2 f2>,<r2 f3>), (<r3 f1>,<r3 f2>,<r3 f3>),...; If the database does not support such statements, it will result  in the error message shown above. It is recommended to verify support for thes

0
blakeorsak
Fan
19 days ago
K
Fan
Karimulla ShaikFan
asked in Community Articles

Read Emails.

Dears,Don’t we have SMTP connector to connect and get the email content like attachments/mail sub/body? 

1
saleempothiwala
Opening Band
25 days ago
K
Fan
kanyewestnewmerch33Fan
asked in Community Articles

How to handle auto increment columns while writing data into MySQL from MongoDB

I discovered that MongoDB lacks a 'id' field, although MySQL has one when transferring data between the two databases. I'm having trouble writing data into MySQL as a result. Regarding this situation, I need help managing the auto-increment column. Can anyone offer any advice or support?I'm also looking into the possibilities of handling this problem with the JDBC Tee processor. Please provide me some tips on how to use the JDBC Tee processor in this situation.

0
K
Fan
27 days ago
Chromeheartshoodie42
Fan
Chromeheartshoodie42Fan
asked in Community Articles

Is it feasible to set up the transformer pipeline so that it runs on an AWS EMR cluster?

I built a primary transformer pipeline that will operate on an EMR cluster. I utilized the essential setup details in Cluster configuration.I'm having the following problem:Pipelinefor a Transformer EngineTransformer 

2
saleempothiwala
Opening Band
29 days ago
Rhudehoodie52
Fan
Rhudehoodie52Fan
asked in Community Articles

Is it feasible to set up the transformer pipeline so that it runs on an AWS EMR cluster?

I built a primary transformer pipeline that will operate on an EMR cluster. I utilized the essential setup details in Cluster configuration.I'm having the following problem:Engine of Transformation  PipelineTransformer 

0
Rhudehoodie52
Fan
29 days ago
Pradeep
StreamSets Employee
PradeepStreamSets Employee
posted in Community Articles

Scenarios to understand data collector job failover and its related configurations

In this article, I will demonstrate example scenarios to help us understand failover retries configurations. When a Data Collector job is enabled for failover, Control Hub retries the failover an infinite number of times by default. However, you may want the failover to stop after a given number of retries. In such cases, you need to define the maximum number of retries to perform.To determine the maximum number of retries, you can configure one or both of the following properties when you configure the job: Failover Retries per Data Collector (FRPD): This refers to the maximum number of pipeline failover retries to attempt on each available Data Collector. The initial start of a pipeline instance on a Data Collector counts as the first retry attempt. Control Hub maintains the failover retry count for each available Data Collector. When a Data Collector reaches the maximum number of failover retries, Control Hub does not attempt to restart additional failed pipelines for the job on tha

0
Pradeep
StreamSets Employee
1 month ago
nicholas.mendoza
Fan
nicholas.mendozaFan
posted in Community Articles

Data Collector failing with "Cannot recover key" error

Issue:Data Collector fails with the following error in the logs: Cannot recover key A possible cause for this error is that the keystore password and the contained key file password do not match. The password configured in https.keystore.password in sdc.properties is used to access the keystore and the key file. If the passwords are different, this leads to not having access to the key file, resulting in the Cannot recover key error. To resolve this, make sure that the keystore file and the contained key file are using the same password.

0
nicholas.mendoza
Fan
1 month ago
rvlozano
StreamSets Employee
rvlozanoStreamSets Employee
asked in Community Articles

Oracle CDC client stage shows only one record when previewing pipeline.

Problem:Oracle CDC client stage shows only one record when previewing. Analysis:Oracle CDC Stage preview is unique from other stage previews that it does not treat a value of 0 or -1 (as forever) when setting the “Batch wait time (secs)”.  According to its mechanics, this is expected and by design. Preview Batch Size is only a limit on the records returned by the pipeline, but, as batches takes precedence. Mitigation:Raise “Batch wait time (secs)” in JDBC Settings to a non-zero value.Lastly, In regards of best practices with Oracle CDC pipelinesA sensible value would be around 15 to 30 seconds. This way you do not produce too small of batches that will impact overall performance. 

0
rvlozano
StreamSets Employee
1 month ago
john.mcavoy
StreamSets Employee
john.mcavoyStreamSets Employee
asked in Community Articles

Oracle Timestamp parsing error in JDBC Query pipelines: ORA-01843

ProblemWhen using a timestamp as an offset in a JDBC Query Consumer origin for an Oracle DB connection, i.e. SELECT * FROM table WHERE modified_timestamp >= '${OFFSET}' ORDER BY modified_timestampThe Oracle JDBC driver is throwing some date string parsing error, e.g.: java.sql.SQLDataException: ORA-01843 BackgroundSDC stores pipeline origin offsets as a Java data with the format: YYYY/MM/DD HH24:Mi:SS, i.e:"2023-04-25 14:34:42"If the Oracle JDBC driver is throwing ORA-01843, this is because the date format used for the offset is not the format it is expecting.If the Oracle session variable: NLS_TIMESTAMP is not equal to YYYY/MM/DD HH24:Mi:SS , the offset date string in the SQL query will not be automatically parsed into a Date type.SolutionTo correct this parsing error, you can do either of the following options:Simply re-write your SQL query to include a to_date to explicitly convert the SDC timestamp offset to Date type, regardless of the Oracle session’s default NLS_TIMESTAMP val

0
john.mcavoy
StreamSets Employee
1 month ago
tamilarasup
Discovered Fame
tamilarasupDiscovered Fame
asked in Community Articles

MERGE

Hi I have 2 tables table one have 3 columns- ID, NAME,LOCATION.table two have 3 columns-ID,LOCATION, EMPLOYMENT TYPEI need to merge the two data table as one after another not a join  and expect output as combine both table in single table likeID, NAME,LOCATION,EMPLOYMENT TYPE 

7
Pradeep
StreamSets Employee
1 month ago
paul
StreamSets Employee
paulStreamSets Employee
asked in Community Articles

Details on SDC Metrics

QuestionHow are the SDC Metrics collected and what do they represent?AnswerThe graphs on the SDC Metrics page in StreamSets Data Collector and the Metrics tab of the Execution Engine page in Control Hub are system-level metrics collected from standard Java core libraries. They should roughly correspond with metrics reported by other system-level tools, such as top and uptime at the command line, as well as external monitoring tools that show system-level metrics. Note that the numbers reported by different tools won’t match each other exactly due to differences in reporting intervals and other factors, but there should be a correlation.

0
paul
StreamSets Employee
1 month ago
blakeorsak
Fan
blakeorsakFan
posted in Community Articles

Job does not stop on error

Problem: When working with jobs, the job does not stop after encountering an error and continues to run. The pipeline is set to retry on error a few times before failing, but never shuts down completely.  Solution:  When creating jobs for pipelines, there are some configuration settings that may override the settings in the pipeline, specifically regarding retries when Failover is enabled. Two job settings in particular, Failover Retries per Data Collector and Global Failover Retries, directly impact this scenario.   Failover Retries per Data Collector is the maximum number of pipeline failover retries to attempt on each available Data Collector. Global Failover Retries is the maximum number of pipeline failover retries to attempt across all available Data Collectors. The default value for these settings is -1, which will result in unlimited retries on the job. Therefore, the pipeline will never completely stop, even if the pipeline settings are configured to do so,  when failover is e

0
blakeorsak
Fan
1 month ago
Pradeep
StreamSets Employee
PradeepStreamSets Employee
posted in Community Articles

Invalid parameter - Date is null TimeNowEL

If you have noticed below log line of ERROR in your SDC or data collector logs, it is possible that the `time:extractStringFromDateTZ` function having a null date object after the evaluation of its first parameter.Invalid parameter - Date is null TimeNowELSolution: This is expected since time:extractStringFromDateTZ function is unaware of what is being evaluated in its parameter. If you are using something like `record:value*` to retrieve the date object and if it happens to be null, the above error is logged. `Returns an empty string when the time zone is not specified or invalid` is mentioned in doc that it needs to be expected.In case if you would like to filter out or categorize the record(s) containing field(s) of null as error, please consider using a stream selector stage to move such records to To Error stage.

0
Pradeep
StreamSets Employee
1 month ago
jqianglee
Fan
jqiangleeFan
asked in Community Articles

how to use jdbc multitable consumer to get result with two different offset from the same table

i hope to use jdbc multitable to gernerate sql like select * from test where create_time > offset order by create_time,   select * from test where update_time > offset order by update_time.one table is ordered by two offset respectively，and data is in a pipelinei add two schema tabs,but i found the pipeline only output data according to the second schema tab config,the first schema config cannot take effct.i do not know whether data according to the second schema tab config overwrite data according to the first

0
jqianglee
Fan
1 month ago
subhanmiq
Fan
subhanmiqFan
asked in Community Articles

Streamsets opensource community support and future plans of SDC usage?

 Quick question:Wanted to check if there is a plan to discontinue open source community support for Streamsets any time soon? Would there be any legal issues if we continue to use sreamsets opensource?

0
subhanmiq
Fan
1 month ago
Mike Calma
StreamSets Employee
Mike CalmaStreamSets Employee
posted in Community Articles

JVM Memory & G1GC

G1GC: Garbage 1st Garbage Collection:Utilizing the G1GC algorithm for the JVM can be extremely helpful in more efficiently managing garbage collection in larger heap size environments. This is especially the case for those attempting to leverage max heap sizes ~6GB or larger as is recommended[4] . G1GC more effectively handles larger heap sizes while minimizing the GC collection latency. This is accomplished through a couple of different methods: defining regions, compacting, and pause prediction. In short, G1GC will divide the memory into regions which are then prioritized for allocation and garbage collection.   GC.log Troubleshooting:When troubleshooting Data Collector and Control Hub memory issues one of the best places to check first is to examine garbage collection health. Garbage collection logging is defined within sdc-env.sh and can be enabled by adding to SDC_JAVA_OPTS with the flag -XX:+PrintGC. Additionally the logs can be located by default under $SDC_HOME/log/. The follow

0
Mike Calma
StreamSets Employee
1 month ago
N
Fan
NileshFan
asked in Community Articles

Oracle CDC to Snowflake while processing CLOB datatype

Hi,Facing error in Data collector pipeline Oracle CDC to Snowflake while processing CLOB datatype table. When inserting value >= 4000 then works fine. But the less than that facing error.SNOWFLAKE_28 - Snowflake MERGE for 'sdc-df26c7ce-fd00-4b5d-a3b1-817b9809e0a3.csv.gz' processed '1' CDC records out of '3'Oracle:create table TEST_LOB_VAL(lobid number(10) primary key, lob_value clob );INSERT INTO TEST_LOB_VAL VALUES (1,to_clob(RPAD('AAAAAAAAAAAAAAAAAA',4000,'A')));INSERT INTO TEST_LOB_VAL VALUES (2,to_clob(RPAD('AAAAAAAAAAAAAAAAAA',2000,'A')));INSERT INTO TEST_LOB_VAL VALUES (3,'AAAAAAAAAA');COMMIT;

4
N
Fan
1 month ago

Badge winners

Show all badges
Terms & Conditions