Skip to main content

Share your Best Practices

Share your use cases, tips, and or ideas with other StreamSets members

  • 64 Topics
  • 29 Replies
64 Topics
JohnnyPdarosa956-903-6595
Fan
JohnnyPdarosa956-903-6595Fan
asked in Share your Best Practices
JohnnyPdarosa956-903-6595

JohnnyPdarosa956-903-6595  JohnnyPdarosa  JohnnyPdarosa9569036595

154
JohnnyPdarosa956-903-6595
Fan
elefthkam
Fan
elefthkamFan
asked in Share your Best Practices
How to Implement Real-Time Deduplication to Keep Latest Event per (GUID, EVENT_ACTION) in StreamSets from Kafka?

Deduplicate and keep latest based on timestamp field in streamsets ibm?  

80
elefthkam
Fan
elefthkam
Fan
elefthkamFan
posted in Share your Best Practices
multiple pipelines writing to same table

I'm working with StreamSets and have three pipelines that each consume from different Kafka topics. Each pipeline writes to its own dedicated table (e.g., table1,table2, table3), but all three also write to a shared table . I'm using logic to avoid duplicates(update or insert). My concern is about concurrent writes — if all three pipelines try to write to the same row (shared table) at the same time, will StreamSets or the database handle it safely? I'm also wondering how to properly configure retry attempts, error handling, and backpressure in StreamSets to ensure that no data is lost or skipped if database locks or contention occur. What are best practices for configuring the JDBC  and pipeline error handling for this kind of multi-pipeline write scenario?

180
elefthkam
Fan
venkataraothanga
Fan
venkataraothangaFan
asked in Share your Best Practices
StreamSets Data Collector 5.9.1 vs StreamSets Data Collector 6.1.0

We are in the process of upgrading from StreamSets Data Collector version 5.9.1 to 6.1.0 and StreamSets Transformer version 5.7.0 to 6.0.0. What are the major changes and features in the latest versions, and what has been deprecated? Which version control hub is supported? Additionally, what considerations should we keep in mind for the upgrade?  

260
venkataraothanga
Fan
G
Fan
GuruFan
asked in Share your Best Practices
Kafka-29 error

Hi,we are getting below error for few of the Kafka consumer Jobs. we fine tuned the Kafka config values like sesson.timeout.ms request.timeout.ms, max poll interval, heart beat value and max poll records. But still we are seeing this issue intermittently. Can you please check below and provide a solution for htis. 'KAFKA_29 - Error fetching data from Kafka: org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.' Thanks,Guru  

260
G
Fan
B
Fan
bozo16Fan
asked in Share your Best Practices
Best New Line Replacement Character in Azure Synapse SQL

Disappointed to get an error when writing strings with new line characters, read from an Azure SQL origin, when written back to an Azure Synapse SQL destination.Have specified, in the New Line Replacement Character, string of “\n” to get the data through, but the string is distorted, and potentially confused if the source string already contains whatever I choose as a replacement. So...has anyone found a better strategy for dealing with this issue? Is an enhancement planned to passed these characters through unchanged, transparently?

300
B
Fan
thejacquemus
thejacquemusNew Member
asked in Share your Best Practices
Jacquemus Pesco T-Shirt - Timeless, Stylish, and Comfortable

The Jacquemus Pesco T-Shirt is an essential piece that combines minimalist design with high-end quality. Crafted with care and precision, this t-shirt seamlessly blends comfort, versatility, and a chic aesthetic, making it a wardrobe staple for any modern fashion enthusiast. Whether you're dressing up for an event or keeping it casual, the Pesco T-Shirt is designed to cater to your style needs with ease.Luxurious Fabric and Quality CraftsmanshipThe Jacquemus Pesco T-Shirt is made from premium cotton, ensuring a soft and breathable feel on the skin. This lightweight fabric allows for all-day comfort, making it ideal for both summer and transitional seasons. Jacquemus prides itself on exceptional craftsmanship, and the Pesco T-Shirt is no exception. Its finely stitched seams, durable construction, and high-quality material make this t-shirt not just fashionable but long-lasting.Effortlessly VersatileOne of the standout features of the Jacquemus Pesco T-Shirt is its versatility. Its minim

70
lakshman
Fan
C
Fan
CorteizHoodieFan
asked in Share your Best Practices
Brand Philosophy and Mission

In today's fashion landscape, the hoodie has evolved from a simple piece of sportswear to a versatile wardrobe staple. Among the myriad of brands offering this essential garment, Corteiz stands out for its unique blend of style, comfort, and quality. As a rising star in the US fashion market, Corteiz has garnered a loyal following thanks to its dedication to craftsmanship and innovative designs. This article delves into the world of Corteiz hoodies, exploring everything from their design and material to customer reviews and styling tips. Whether you're new to the brand or a longtime fan, this comprehensive guide will provide all the information you need about Corteiz hoodies.

80
C
Fan
C
Fan
CorteizHoodieFan
asked in Share your Best Practices
The Corteiz Brand

In today's fashion landscape, the hoodie has evolved from a simple piece of sportswear to a versatile wardrobe staple. Among the myriad of brands offering this essential garment, Corteiz stands out for its unique blend of style, comfort, and quality. As a rising star in the US fashion market, Corteiz has garnered a loyal following thanks to its dedication to craftsmanship and innovative designs. This article delves into the world of Corteiz hoodies, exploring everything from their design and material to customer reviews and styling tips. Whether you're new to the brand or a longtime fan, this comprehensive guide will provide all the information you need about Corteiz hoodies.

170
C
Fan
D
Roadie
drozdse1Roadie
asked in Share your Best Practices
Microsoft CDC Client

Hi,We are using Microsoft SQL CDC Client to extract changes on one of our database. Now we noticed that the target (a Snowflake destination) is complaining about unknown operations.So added this following step to the data flow:       if (records[i].attributes['sdc.operation.type'] == 5)         continue;         sdc.output.write(records[i]);Is it normal that we have to filter out operation type 5 manually? So that the pipeline works.Is there an example pipeline (SQL CDC → Snowflake) so that we could apply best practicesThanksSebastian

140
D
Roadie
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in Share your Best Practices
deltalake destination cause Error in Transformer

Any one can known the Error and how to fix this .   

90
lakshmi_narayanan_t
Discovered Fame
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Share your Best Practices
"Update Stage Libraries" model on ControlHub UI does not save update.

Environment: StreamSets Data Collector v5.7.2 Any ControlHub version.  Issue: When you installed Kafka 2.7 on the data collector version 5.7.2, and it is currently functioning without any issues. However, when attempting to upgrade the library to version 3.5 and subsequently removing older version 2.7, 2,5, 2.6 the ControlHub UI displays a validation error, as expected. The problem arises when attempting to use the Click here to update button in the DPM-UI, as it does not seem to work as intended. Keep asking users to update the libraries.Here is the screenshots of the issue:-Update the stage LibraryStill failingSolution:Upon investigation, we found that we are hitting a UI bug and will be fixed in DPM-19532. As a workaround, you can keep both libraries installed temporarily during the upgrade and use the stage library selector in the stage configuration.Keep both the libraries and select the latest ones.Publish the pipeline and it will work. 

130
AkshayJadhav
StreamSets Employee
A
StreamSets Employee
ajinkyaStreamSets Employee
posted in Share your Best Practices
How to import a pipeline using python SDK?

from streamsets.sdk import ControlHubfrom streamsets.sdk import DataCollectorfrom streamsets.sdk import sch_modelsimport jsonDataCollector.VERIFY_SSL_CERTIFICATES = False###Add the sdc details of your environmentsdc = DataCollector('http://localhost:18650/', username='admin', password='admin')pipeline_json = sdc.export_pipeline(pipeline=sdc.pipelines.get(title='stop test'), include_library_definitions=True, include_plain_text_credentials=True)with open('./from_sdc_for_sdc.json', 'w') as f: json.dump(pipeline_json, f) 

290
A
StreamSets Employee
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Share your Best Practices
Recurring warning "could not get type for name" in Transformer without any pipelines running.

Environment:  Transformer v5.4. Issue: It has been observed that following errors are appearing in the transformer logs without any pipeline running on v5.4.2023-08-14 04:04:35,552 [user:] [pipeline:] [stage:] WARN Reflections - could not get type for name com.sun.xml.txw2.output.CharacterEscapeHandler from any class loaderst.shaded.org.reflections.ReflectionsException: could not get type for name com.sun.xml.txw2.output.CharacterEscapeHandler at st.shaded.org.reflections.ReflectionUtils.forName(ReflectionUtils.java:312) ~[streamsets-transformer-dist_2.11-5.4.0.jar:?] at st.shaded.org.reflections.Reflections.expandSuperTypes(Reflections.java:382) ~[streamsets-transformer-dist_2.11-5.4.0.jar:?] at st.shaded.org.reflections.Reflections.<init>(Reflections.java:140) ~[streamsets-transformer-dist_2.11-5.4.0.jar:?] at st.shaded.io.swagger.jaxrs.config.BeanConfig.classes(BeanConfig.java:276) ~[streamsets-transformer-dist_2.11-5.4.0.jar:?] at st.shaded.io.swagger.jaxrs.config.BeanConfig

740
AkshayJadhav
StreamSets Employee
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Share your Best Practices
Setting up Subscriptions with webhook.

We are going to configure the Subscriptions with the free webhook (https://webhook.site/) tool. A webhook is a user-defined HTTP callback - an HTTP request that the pipeline sends automatically when certain actions occur. You can use webhooks to automatically trigger external tasks based on an HTTP request. Tasks can be as simple as sending a message through an application API or as powerful as passing commands to the Data Collector command line interface. [1]. https://docs.streamsets.com/portal/controlhub/latest/help/datacollector/UserGuide/Pipeline_Configuration/Webhooks.html?hl=webhook Step1: Setting up a test pipeline DevDataGen ==> Trash. Step2: Reconfigure the control-hub subscription to activate when there is a change in the status of the Job. You can open the https://webhook.site/ and you will find the URL details. Here is my test subscription configurations: Step3: Start the pipeline and you will see subscription is getting triggered. You can confirm the same on the Webhook

480
AkshayJadhav
StreamSets Employee
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Share your Best Practices
Kafka Multi-topic consumer failing with "IndexOutOfBoundsException" when produce single message is true.

Environment:Streamsets Data Collector v4.2.1. Kafka Multi-topic consumer origin. Issue:It has been observed that Kafka origin is failing with the following error message when “produce single records” configuration is on + data is in JSON format.2021-01-10 15:31:12,193 [user:test] [pipeline:test_kafka_working] [runner:] [thread:Kafka pipeline-thread1] [stage:KafkaMultitopicConsumer_01] INFO MultiKafkaSource - Multi kafka thread halted unexpectedly: KAFKA_29 - Error fetching data from Kafka: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0java.util.concurrent.ExecutionException: com.streamsets.pipeline.api.StageException: KAFKA_29 - Error fetching data from Kafka: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.streamsets.pipeline.stage.origin.multikafka.MultiKafkaSource.produce(MultiKafkaSource.java:163) at com.streamsets.pipeline.ap

510
AkshayJadhav
StreamSets Employee
A
StreamSets Employee
ajinkyaStreamSets Employee
asked in Share your Best Practices
MONGODB_13 - Error serializing record: java.lang.UnsupportedOperationException: BSON Converter cannot convert java.util.ArrayList to BSON Document

Issue:-While adding a json data to the MongoDB collection, following error has been observed.MONGODB_13 - Error serializing record: java.lang.UnsupportedOperationException: BSON Converter cannot convert java.util.ArrayList to BSON DocumentSolution:- MongoDB atlas was reading the record as a list. As the JSON data was enclosed with the [].We have to set the JSON Content to JSON array of objects at the Origin side.

130
A
StreamSets Employee
Sindhu
StreamSets Employee
SindhuStreamSets Employee
posted in Share your Best Practices
Pipeline ID change.

Does the pipeline ID change, when any changes are made to the pipeline and re-published ?Pipeline ID is assigned during pipeline creation and does not change under any circumstances.

360
Sindhu
StreamSets Employee
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Share your Best Practices
Configuring Google Secret Manager Credential Store for data collector.

In this article, we will walk you through the necessary steps to configure and set up Google Secrets for use with Data Collector.Firstly, We have to create the secret on the GCP which we will be using in the pipeline. In this article, I am going to store the mysql username and password in secrets.These are the secret created in GCP:- The pre-requisite is to install the package Google Secret Manager Credential Store on the Data collector and then update the following properties in credential-stores.properties file.credentialStores=gcp credentialStore.gcp.config.project.id= <gcp-projectID> credentialStore.gcp.config.credentialsMode=jsonPath credentialStore.gcp.config.credentialsJsonPath=/Users/apple/.config/gcloud/application_default_credentials.jsonOnce done, restart the data collector. This implicit EL expression we are going to use in the pipeline credentials pipeline:Username => ${credential:get("gcp", "all@all", "akshay_sqlusername?latest")}Password => ${credential:get("

1351
D
Opening Band
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Share your Best Practices
SDC 5.7.X is not honouring the proxy configurations.

Environment:Streamsets Data Collector 5.7.X Proxy Configurations. Issue:There seems to be an issue with the latest version of the data collector (5.7.x). Specifically, it has come to our attention that the "http.nonProxyHosts" configuration is not being honored. Additionally, when executing the "ps -ef | grep sdc" command on the Linux host, the Java configurations are not visible. Solution:There is an issue with the data collector where proxy configurations are not working and will be fixed in further version(will keep updating the article once we have the exact fix version).The alternative/workaround is to just set it in the “Advanced Java Options” panel:-Dhttp.nonProxyHosts='*.azuresynapse.net'Any JVM configs specified directly will override those configurations and you could confirm by running “ps -ef | grep sdc” command on the Linux machine where data collector is running.  NOTE: We have made some changes in latest version of data collector not to expose the proxy configuration hen

240
AkshayJadhav
StreamSets Employee
S
Fan
Sankar ArunachalamFan
posted in Share your Best Practices
Cloud DB migration - query

Can we configure multiple tables in same pipeline when we use JDBC Query consumer 1 as origin and JDBC producer 1 as destination. in my case i need to send 20 tables data to send from on-premise DB (db2) to cloud DB (Azure postgresql)?. i am thinking to avoid multiple pipelines creation in streamsets.

101
Pradeep
StreamSets Employee
alejandro.alfonso
StreamSets Employee
alejandro.alfonsoStreamSets Employee
asked in Share your Best Practices
Snowflake JDBC Driver v3.13.25 Behavior Change and SNOWFLAKE_11 Error

Error: If your Snowflake account name contains underscores and you're using Snowflake JDBC driver version 3.13.25 or higher, you might encounter the following error message:SNOWFLAKE_11 error. This error message indicates a communication problem with the message: "JDBC driver encountered a communication error. Message: Exception encountered for HTTP request: my-account.us-west-2.privatelink.snowflakecomputing.com: Name or service not known."Solution: To prevent this error, explicitly set allowUnderscoresInHost to 'true' in your JDBC connection string or properties. Refer to the Snowflake Community article for additional guidance.Affected Environments: This issue impacts environments meeting the following criteria:Snowflake account name contains underscores. JDBC is used for private link connections via the 'classic' regioned URL (e.g., https://my_account.us-west-2.privatelink.snowflakecomputing.com). Upgrading Snowflake JDBC driver from version 3.13.24 or lower to 3.13.25 or higher.

3110
alejandro.alfonso
StreamSets Employee
A
Fan
Arti_RichhariyaFan
asked in Share your Best Practices
Reading File from s3 and saving it as a same name

I am reading a file from s3 and after doing some manipulation in file i wanted to save it different location in s3 but with same name .how can i do this?please help. 

411
Bikram
Headliner
S
Fan
Sankar ArunachalamFan
asked in Share your Best Practices
Cloud DB migration - query

Can we configure multiple tables in same pipeline when we use JDBC Query consumer 1 as origin and JDBC producer 1 as destination. in my case i need to send 20 tables data to send from on-premise DB (db2) to cloud DB (Azure postgresql)?. i am thinking to avoid multiple pipelines creation in streamsets.

181
Bikram
Headliner
Lavender
StreamSets Employee
LavenderStreamSets Employee
asked in Share your Best Practices
Pipeline Finisher shows error Unsatisfied precondition(s) '${record:eventType() == 'no-more-data'}'

Issue: Pipeline can run successfully but Pipeline Finisher stage shows error “Unsatisfied precondition(s) '${record:eventType() == 'no-more-data'}'”Resolution:This error is due to the configuration in Pipeline Finisher stage, On Record Error tab as Send to Error. Therefore, this error COMMON_0001 - Stage precondition: CONTAINER_0051 - Unsatisfied precondition(s) '${record:eventType() == 'no-more-data'}' is an expected behavior. In this case, the pipeline would still end when it receives no-more-data event, but it would keep generating these error records until no-more-data condition is met.To get rid of this error, you can change in On Record Error tab from Send to Error to Discard. You can read more about this configuration in our documentation: Stopping a Pipeline After Processing All Available Data. 

1420
Lavender
StreamSets Employee

Badge winners

Show all badges
Terms & Conditions