Skip to main content

Share your Best Practices

Share your use cases, tips, and or ideas with other StreamSets members

  • 60 Topics
  • 25 Replies
60 Topics
G
Fan
GuruFan
asked in Share your Best Practices
Kafka-29 error

Hi,we are getting below error for few of the Kafka consumer Jobs. we fine tuned the Kafka config values like sesson.timeout.ms request.timeout.ms, max poll interval, heart beat value and max poll records. But still we are seeing this issue intermittently. Can you please check below and provide a solution for htis. 'KAFKA_29 - Error fetching data from Kafka: org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.' Thanks,Guru  

70
G
Fan
17 days ago
B
Fan
bozo16Fan
asked in Share your Best Practices
Best New Line Replacement Character in Azure Synapse SQL

Disappointed to get an error when writing strings with new line characters, read from an Azure SQL origin, when written back to an Azure Synapse SQL destination.Have specified, in the New Line Replacement Character, string of “\n” to get the data through, but the string is distorted, and potentially confused if the source string already contains whatever I choose as a replacement. So...has anyone found a better strategy for dealing with this issue? Is an enhancement planned to passed these characters through unchanged, transparently?

260
B
Fan
2 months ago
thejacquemus
thejacquemusNew Member
asked in Share your Best Practices
Jacquemus Pesco T-Shirt - Timeless, Stylish, and Comfortable

The Jacquemus Pesco T-Shirt is an essential piece that combines minimalist design with high-end quality. Crafted with care and precision, this t-shirt seamlessly blends comfort, versatility, and a chic aesthetic, making it a wardrobe staple for any modern fashion enthusiast. Whether you're dressing up for an event or keeping it casual, the Pesco T-Shirt is designed to cater to your style needs with ease.Luxurious Fabric and Quality CraftsmanshipThe Jacquemus Pesco T-Shirt is made from premium cotton, ensuring a soft and breathable feel on the skin. This lightweight fabric allows for all-day comfort, making it ideal for both summer and transitional seasons. Jacquemus prides itself on exceptional craftsmanship, and the Pesco T-Shirt is no exception. Its finely stitched seams, durable construction, and high-quality material make this t-shirt not just fashionable but long-lasting.Effortlessly VersatileOne of the standout features of the Jacquemus Pesco T-Shirt is its versatility. Its minim

60
thejacquemus
3 months ago
C
Fan
CorteizHoodieFan
asked in Share your Best Practices
Brand Philosophy and Mission

In today's fashion landscape, the hoodie has evolved from a simple piece of sportswear to a versatile wardrobe staple. Among the myriad of brands offering this essential garment, Corteiz stands out for its unique blend of style, comfort, and quality. As a rising star in the US fashion market, Corteiz has garnered a loyal following thanks to its dedication to craftsmanship and innovative designs. This article delves into the world of Corteiz hoodies, exploring everything from their design and material to customer reviews and styling tips. Whether you're new to the brand or a longtime fan, this comprehensive guide will provide all the information you need about Corteiz hoodies.

70
C
Fan
5 months ago
C
Fan
CorteizHoodieFan
asked in Share your Best Practices
The Corteiz Brand

In today's fashion landscape, the hoodie has evolved from a simple piece of sportswear to a versatile wardrobe staple. Among the myriad of brands offering this essential garment, Corteiz stands out for its unique blend of style, comfort, and quality. As a rising star in the US fashion market, Corteiz has garnered a loyal following thanks to its dedication to craftsmanship and innovative designs. This article delves into the world of Corteiz hoodies, exploring everything from their design and material to customer reviews and styling tips. Whether you're new to the brand or a longtime fan, this comprehensive guide will provide all the information you need about Corteiz hoodies.

110
C
Fan
5 months ago
D
Roadie
drozdse1Roadie
asked in Share your Best Practices
Microsoft CDC Client

Hi,We are using Microsoft SQL CDC Client to extract changes on one of our database. Now we noticed that the target (a Snowflake destination) is complaining about unknown operations.So added this following step to the data flow:       if (records[i].attributes['sdc.operation.type'] == 5)         continue;         sdc.output.write(records[i]);Is it normal that we have to filter out operation type 5 manually? So that the pipeline works.Is there an example pipeline (SQL CDC → Snowflake) so that we could apply best practicesThanksSebastian

110
D
Roadie
9 months ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in Share your Best Practices
deltalake destination cause Error in Transformer

Any one can known the Error and how to fix this .   

90
lakshmi_narayanan_t
Discovered Fame
1 year ago
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Share your Best Practices
"Update Stage Libraries" model on ControlHub UI does not save update.

Environment: StreamSets Data Collector v5.7.2 Any ControlHub version.  Issue: When you installed Kafka 2.7 on the data collector version 5.7.2, and it is currently functioning without any issues. However, when attempting to upgrade the library to version 3.5 and subsequently removing older version 2.7, 2,5, 2.6 the ControlHub UI displays a validation error, as expected. The problem arises when attempting to use the Click here to update button in the DPM-UI, as it does not seem to work as intended. Keep asking users to update the libraries.Here is the screenshots of the issue:-Update the stage LibraryStill failingSolution:Upon investigation, we found that we are hitting a UI bug and will be fixed in DPM-19532. As a workaround, you can keep both libraries installed temporarily during the upgrade and use the stage library selector in the stage configuration.Keep both the libraries and select the latest ones.Publish the pipeline and it will work. 

100
AkshayJadhav
StreamSets Employee
1 year ago
A
StreamSets Employee
ajinkyaStreamSets Employee
posted in Share your Best Practices
How to import a pipeline using python SDK?

from streamsets.sdk import ControlHubfrom streamsets.sdk import DataCollectorfrom streamsets.sdk import sch_modelsimport jsonDataCollector.VERIFY_SSL_CERTIFICATES = False###Add the sdc details of your environmentsdc = DataCollector('http://localhost:18650/', username='admin', password='admin')pipeline_json = sdc.export_pipeline(pipeline=sdc.pipelines.get(title='stop test'), include_library_definitions=True, include_plain_text_credentials=True)with open('./from_sdc_for_sdc.json', 'w') as f: json.dump(pipeline_json, f) 

200
A
StreamSets Employee
1 year ago
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Share your Best Practices
Recurring warning "could not get type for name" in Transformer without any pipelines running.

Environment:  Transformer v5.4. Issue: It has been observed that following errors are appearing in the transformer logs without any pipeline running on v5.4.2023-08-14 04:04:35,552 [user:] [pipeline:] [stage:] WARN Reflections - could not get type for name com.sun.xml.txw2.output.CharacterEscapeHandler from any class loaderst.shaded.org.reflections.ReflectionsException: could not get type for name com.sun.xml.txw2.output.CharacterEscapeHandler at st.shaded.org.reflections.ReflectionUtils.forName(ReflectionUtils.java:312) ~[streamsets-transformer-dist_2.11-5.4.0.jar:?] at st.shaded.org.reflections.Reflections.expandSuperTypes(Reflections.java:382) ~[streamsets-transformer-dist_2.11-5.4.0.jar:?] at st.shaded.org.reflections.Reflections.<init>(Reflections.java:140) ~[streamsets-transformer-dist_2.11-5.4.0.jar:?] at st.shaded.io.swagger.jaxrs.config.BeanConfig.classes(BeanConfig.java:276) ~[streamsets-transformer-dist_2.11-5.4.0.jar:?] at st.shaded.io.swagger.jaxrs.config.BeanConfig

540
AkshayJadhav
StreamSets Employee
1 year ago
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Share your Best Practices
Setting up Subscriptions with webhook.

We are going to configure the Subscriptions with the free webhook (https://webhook.site/) tool. A webhook is a user-defined HTTP callback - an HTTP request that the pipeline sends automatically when certain actions occur. You can use webhooks to automatically trigger external tasks based on an HTTP request. Tasks can be as simple as sending a message through an application API or as powerful as passing commands to the Data Collector command line interface. [1]. https://docs.streamsets.com/portal/controlhub/latest/help/datacollector/UserGuide/Pipeline_Configuration/Webhooks.html?hl=webhook Step1: Setting up a test pipeline DevDataGen ==> Trash. Step2: Reconfigure the control-hub subscription to activate when there is a change in the status of the Job. You can open the https://webhook.site/ and you will find the URL details. Here is my test subscription configurations: Step3: Start the pipeline and you will see subscription is getting triggered. You can confirm the same on the Webhook

310
AkshayJadhav
StreamSets Employee
1 year ago
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Share your Best Practices
Kafka Multi-topic consumer failing with "IndexOutOfBoundsException" when produce single message is true.

Environment:Streamsets Data Collector v4.2.1. Kafka Multi-topic consumer origin. Issue:It has been observed that Kafka origin is failing with the following error message when “produce single records” configuration is on + data is in JSON format.2021-01-10 15:31:12,193 [user:test] [pipeline:test_kafka_working] [runner:] [thread:Kafka pipeline-thread1] [stage:KafkaMultitopicConsumer_01] INFO MultiKafkaSource - Multi kafka thread halted unexpectedly: KAFKA_29 - Error fetching data from Kafka: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0java.util.concurrent.ExecutionException: com.streamsets.pipeline.api.StageException: KAFKA_29 - Error fetching data from Kafka: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.streamsets.pipeline.stage.origin.multikafka.MultiKafkaSource.produce(MultiKafkaSource.java:163) at com.streamsets.pipeline.ap

320
AkshayJadhav
StreamSets Employee
1 year ago
A
StreamSets Employee
ajinkyaStreamSets Employee
asked in Share your Best Practices
MONGODB_13 - Error serializing record: java.lang.UnsupportedOperationException: BSON Converter cannot convert java.util.ArrayList to BSON Document

Issue:-While adding a json data to the MongoDB collection, following error has been observed.MONGODB_13 - Error serializing record: java.lang.UnsupportedOperationException: BSON Converter cannot convert java.util.ArrayList to BSON DocumentSolution:- MongoDB atlas was reading the record as a list. As the JSON data was enclosed with the [].We have to set the JSON Content to JSON array of objects at the Origin side.

120
A
StreamSets Employee
1 year ago
Sindhu
StreamSets Employee
SindhuStreamSets Employee
posted in Share your Best Practices
Pipeline ID change.

Does the pipeline ID change, when any changes are made to the pipeline and re-published ?Pipeline ID is assigned during pipeline creation and does not change under any circumstances.

350
Sindhu
StreamSets Employee
1 year ago
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Share your Best Practices
Configuring Google Secret Manager Credential Store for data collector.

In this article, we will walk you through the necessary steps to configure and set up Google Secrets for use with Data Collector.Firstly, We have to create the secret on the GCP which we will be using in the pipeline. In this article, I am going to store the mysql username and password in secrets.These are the secret created in GCP:- The pre-requisite is to install the package Google Secret Manager Credential Store on the Data collector and then update the following properties in credential-stores.properties file.credentialStores=gcp credentialStore.gcp.config.project.id= <gcp-projectID> credentialStore.gcp.config.credentialsMode=jsonPath credentialStore.gcp.config.credentialsJsonPath=/Users/apple/.config/gcloud/application_default_credentials.jsonOnce done, restart the data collector. This implicit EL expression we are going to use in the pipeline credentials pipeline:Username => ${credential:get("gcp", "all@all", "akshay_sqlusername?latest")}Password => ${credential:get("

1221
D
Opening Band
1 year ago
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Share your Best Practices
SDC 5.7.X is not honouring the proxy configurations.

Environment:Streamsets Data Collector 5.7.X Proxy Configurations. Issue:There seems to be an issue with the latest version of the data collector (5.7.x). Specifically, it has come to our attention that the "http.nonProxyHosts" configuration is not being honored. Additionally, when executing the "ps -ef | grep sdc" command on the Linux host, the Java configurations are not visible. Solution:There is an issue with the data collector where proxy configurations are not working and will be fixed in further version(will keep updating the article once we have the exact fix version).The alternative/workaround is to just set it in the “Advanced Java Options” panel:-Dhttp.nonProxyHosts='*.azuresynapse.net'Any JVM configs specified directly will override those configurations and you could confirm by running “ps -ef | grep sdc” command on the Linux machine where data collector is running.  NOTE: We have made some changes in latest version of data collector not to expose the proxy configuration hen

220
AkshayJadhav
StreamSets Employee
1 year ago
S
Fan
Sankar ArunachalamFan
posted in Share your Best Practices
Cloud DB migration - query

Can we configure multiple tables in same pipeline when we use JDBC Query consumer 1 as origin and JDBC producer 1 as destination. in my case i need to send 20 tables data to send from on-premise DB (db2) to cloud DB (Azure postgresql)?. i am thinking to avoid multiple pipelines creation in streamsets.

101
Pradeep
StreamSets Employee
1 year ago
alejandro.alfonso
StreamSets Employee
alejandro.alfonsoStreamSets Employee
asked in Share your Best Practices
Snowflake JDBC Driver v3.13.25 Behavior Change and SNOWFLAKE_11 Error

Error: If your Snowflake account name contains underscores and you're using Snowflake JDBC driver version 3.13.25 or higher, you might encounter the following error message:SNOWFLAKE_11 error. This error message indicates a communication problem with the message: "JDBC driver encountered a communication error. Message: Exception encountered for HTTP request: my-account.us-west-2.privatelink.snowflakecomputing.com: Name or service not known."Solution: To prevent this error, explicitly set allowUnderscoresInHost to 'true' in your JDBC connection string or properties. Refer to the Snowflake Community article for additional guidance.Affected Environments: This issue impacts environments meeting the following criteria:Snowflake account name contains underscores. JDBC is used for private link connections via the 'classic' regioned URL (e.g., https://my_account.us-west-2.privatelink.snowflakecomputing.com). Upgrading Snowflake JDBC driver from version 3.13.24 or lower to 3.13.25 or higher.

2260
alejandro.alfonso
StreamSets Employee
1 year ago
A
Fan
Arti_RichhariyaFan
asked in Share your Best Practices
Reading File from s3 and saving it as a same name

I am reading a file from s3 and after doing some manipulation in file i wanted to save it different location in s3 but with same name .how can i do this?please help. 

321
Bikram
Headliner
1 year ago
S
Fan
Sankar ArunachalamFan
asked in Share your Best Practices
Cloud DB migration - query

Can we configure multiple tables in same pipeline when we use JDBC Query consumer 1 as origin and JDBC producer 1 as destination. in my case i need to send 20 tables data to send from on-premise DB (db2) to cloud DB (Azure postgresql)?. i am thinking to avoid multiple pipelines creation in streamsets.

171
Bikram
Headliner
1 year ago
Lavender
StreamSets Employee
LavenderStreamSets Employee
asked in Share your Best Practices
Pipeline Finisher shows error Unsatisfied precondition(s) '${record:eventType() == 'no-more-data'}'

Issue: Pipeline can run successfully but Pipeline Finisher stage shows error “Unsatisfied precondition(s) '${record:eventType() == 'no-more-data'}'”Resolution:This error is due to the configuration in Pipeline Finisher stage, On Record Error tab as Send to Error. Therefore, this error COMMON_0001 - Stage precondition: CONTAINER_0051 - Unsatisfied precondition(s) '${record:eventType() == 'no-more-data'}' is an expected behavior. In this case, the pipeline would still end when it receives no-more-data event, but it would keep generating these error records until no-more-data condition is met.To get rid of this error, you can change in On Record Error tab from Send to Error to Discard. You can read more about this configuration in our documentation: Stopping a Pipeline After Processing All Available Data. 

1140
Lavender
StreamSets Employee
1 year ago
alejandro.alfonso
StreamSets Employee
alejandro.alfonsoStreamSets Employee
posted in Share your Best Practices
Legacy Control Hub & SAML - WEB_SSO_204 error

Problem:When attempting to log in to Legacy Control Hub with SAML enabled, users encounter an HTTP 401 error after trying to authenticate with the Identity Provider. The Login Audit trail provides the following information:HTTP Status Code: 401 Error Type: WEB_SSO_204 Response Status: urn:oasis:names:tc:SAML:2.0:status:RequesterSolution:The above error basically means that the Identity Provider has not granted authorization to the user, resulting in a 401 error response to Legacy Control Hub.This issue may stem from a misconfiguration of SAML settings between Legacy Control Hub and the Identity Provider. In most cases, the problem is related to the SAML keys used by Control Hub. Ensure that the public keys are valid and that the Identity Provider is utilizing the same keys. We recommend exchanging fresh copies of metadata and public keys between Legacy Control Hub and the Identity Provider. Afterward, clear your browser's cache and attempt to log in to Control Hub again. If you encount

90
alejandro.alfonso
StreamSets Employee
1 year ago
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Share your Best Practices
Error "DELTA_PATH_DOES_NOT_EXIST" while running Databricks Delta Lake pipeline.

Environment:  Data Collector. Destination Databricks Delta Lake. Issue:  The pipeline is encountering a "DELTA_PATH_DOES_NOT_EXIST" error, signaling that the specified path does not exist. To resolve this, we have manually created the file in Databricks.DELTA_LAKE_37 - Runnable process batch failed: DELTA_LAKE_12 - Could not perform SQL operation: java.sql.SQLException: [Databricks][DatabricksJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: org.apache.hive.service.cli.HiveSQLException: Error running query: [DELTA_PATH_DOES_NOT_EXIST] org.apache.spark.sql.AnalysisException: dbfs:/dash/DeltaLake/TEST doesn't exist at org.apache.spark.sql.hive.thriftserver.HiveThriftServerErrors$.runningQueryError(HiveThriftServerErrors.scala:48) at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.$anonfun$execute$1(SparkExecuteStatementOperation.scala:694) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at com.databricks.unit

4310
AkshayJadhav
StreamSets Employee
1 year ago
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Share your Best Practices
How to connect to Databricks and copy data into Delta Lake tables?

In this article, we will explore the process of creating a pipeline to transfer data from a MySQL database to a Delta Lake tables on Databricks.Environment:Data Collector on DataOps platform. Destination Databricks Delta Lake.These are the steps we need to follow to develop the pipeline: Login to Databricks UI » Compute » Create new compute then fill out all the details such as spark version, type of cluster, worker type. Once everything is finalised, click on create compute.Creating compute will take some sweet time.Now, we will have to fetch the following information from the cluster and populate into the pipeline configuration section. Token: In your Databricks workspace, click your Databricks username in the top bar, and then select User Settings from the dropdown » Develop » Token » Generate a new token. JDBC String: Click on the Databricks newly created cluster, scroll down and expand Advance Option tab » JDBC/ODBC. For staging purpose, we need to use AWS S3, Google Storage or

290
AkshayJadhav
StreamSets Employee
1 year ago
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
asked in Share your Best Practices
Configuring Azure Key Vault Credentials Store for data collector.

In this article, we will walk you through the necessary steps to configure and set up Azure Secrets for use with Data Collector.Firstly, We have to create the secret on the Azure portal which we will be using in the pipeline. In this article, I am going to store the mysql username and password in secrets.  The pre-requisite is to install the package Azure Key Vault Credentials Store on the Data collector and then update the following properties in credential-stores.properties file.credentialStores=azure credentialStore.azure.config.vault.url=https://<YOUR_KEY_VAULT>.vault.azure.net/ credentialStore.azure.config.client.id=<MUST BE SET> credentialStore.azure.config.client.key=<MUST BE SET>This is how you can create the secret and add the values to it:-Secret creating wizard. Added 2 secret.Once done, restart the data collector and add the following EL expression in the pipeline to fetch the username and password.Username => ${credential:get("azure", "all@all", "SQLus

790
AkshayJadhav
StreamSets Employee
1 year ago

Badge winners

Show all badges
Terms & Conditions