30-Day Free Trial: It’s Never Been Easier To Get Started With StreamSets

11 months ago

Home
Community overview
StreamSets Platform
Share your Best Practices

Share your Best Practices

Share your use cases, tips, and or ideas with other StreamSets members

57 Topics
25 Replies

When you subscribe we will email you when there is a new topic in this category

57 Topics

Recently active

Most replies Most views Newest first

CorteizHoodieFan

asked in Share your Best Practices

AkshayJadhavStreamSets Employee

posted in Share your Best Practices

"Update Stage Libraries" model on ControlHub UI does not save update.

Environment: StreamSets Data Collector v5.7.2 Any ControlHub version. Issue: When you installed Kafka 2.7 on the data collector version 5.7.2, and it is currently functioning without any issues. However, when attempting to upgrade the library to version 3.5 and subsequently removing older version 2.7, 2,5, 2.6 the ControlHub UI displays a validation error, as expected. The problem arises when attempting to use the Click here to update button in the DPM-UI, as it does not seem to work as intended. Keep asking users to update the libraries.Here is the screenshots of the issue:-Update the stage LibraryStill failingSolution:Upon investigation, we found that we are hitting a UI bug and will be fixed in DPM-19532. As a workaround, you can keep both libraries installed temporarily during the upgrade and use the stage library selector in the stage configuration.Keep both the libraries and select the latest ones.Publish the pipeline and it will work.

7 months ago

ajinkyaStreamSets Employee

posted in Share your Best Practices

How to import a pipeline using python SDK?

from streamsets.sdk import ControlHubfrom streamsets.sdk import DataCollectorfrom streamsets.sdk import sch_modelsimport jsonDataCollector.VERIFY_SSL_CERTIFICATES = False###Add the sdc details of your environmentsdc = DataCollector('http://localhost:18650/', username='admin', password='admin')pipeline_json = sdc.export_pipeline(pipeline=sdc.pipelines.get(title='stop test'), include_library_definitions=True, include_plain_text_credentials=True)with open('./from_sdc_for_sdc.json', 'w') as f: json.dump(pipeline_json, f)

7 months ago

AkshayJadhavStreamSets Employee

posted in Share your Best Practices

Recurring warning "could not get type for name" in Transformer without any pipelines running.

Environment: Transformer v5.4. Issue: It has been observed that following errors are appearing in the transformer logs without any pipeline running on v5.4.2023-08-14 04:04:35,552 [user:] [pipeline:] [stage:] WARN Reflections - could not get type for name com.sun.xml.txw2.output.CharacterEscapeHandler from any class loaderst.shaded.org.reflections.ReflectionsException: could not get type for name com.sun.xml.txw2.output.CharacterEscapeHandler at st.shaded.org.reflections.ReflectionUtils.forName(ReflectionUtils.java:312) ~[streamsets-transformer-dist_2.11-5.4.0.jar:?] at st.shaded.org.reflections.Reflections.expandSuperTypes(Reflections.java:382) ~[streamsets-transformer-dist_2.11-5.4.0.jar:?] at st.shaded.org.reflections.Reflections.<init>(Reflections.java:140) ~[streamsets-transformer-dist_2.11-5.4.0.jar:?] at st.shaded.io.swagger.jaxrs.config.BeanConfig.classes(BeanConfig.java:276) ~[streamsets-transformer-dist_2.11-5.4.0.jar:?] at st.shaded.io.swagger.jaxrs.config.BeanConfig

7 months ago

AkshayJadhavStreamSets Employee

posted in Share your Best Practices

Setting up Subscriptions with webhook.

We are going to configure the Subscriptions with the free webhook (https://webhook.site/) tool. A webhook is a user-defined HTTP callback - an HTTP request that the pipeline sends automatically when certain actions occur. You can use webhooks to automatically trigger external tasks based on an HTTP request. Tasks can be as simple as sending a message through an application API or as powerful as passing commands to the Data Collector command line interface. [1]. https://docs.streamsets.com/portal/controlhub/latest/help/datacollector/UserGuide/Pipeline_Configuration/Webhooks.html?hl=webhook Step1: Setting up a test pipeline DevDataGen ==> Trash. Step2: Reconfigure the control-hub subscription to activate when there is a change in the status of the Job. You can open the https://webhook.site/ and you will find the URL details. Here is my test subscription configurations: Step3: Start the pipeline and you will see subscription is getting triggered. You can confirm the same on the Webhook

7 months ago

AkshayJadhavStreamSets Employee

posted in Share your Best Practices

Kafka Multi-topic consumer failing with "IndexOutOfBoundsException" when produce single message is true.

Environment:Streamsets Data Collector v4.2.1. Kafka Multi-topic consumer origin. Issue:It has been observed that Kafka origin is failing with the following error message when “produce single records” configuration is on + data is in JSON format.2021-01-10 15:31:12,193 [user:test] [pipeline:test_kafka_working] [runner:] [thread:Kafka pipeline-thread1] [stage:KafkaMultitopicConsumer_01] INFO MultiKafkaSource - Multi kafka thread halted unexpectedly: KAFKA_29 - Error fetching data from Kafka: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0java.util.concurrent.ExecutionException: com.streamsets.pipeline.api.StageException: KAFKA_29 - Error fetching data from Kafka: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.streamsets.pipeline.stage.origin.multikafka.MultiKafkaSource.produce(MultiKafkaSource.java:163) at com.streamsets.pipeline.ap

7 months ago

ajinkyaStreamSets Employee

asked in Share your Best Practices

MONGODB_13 - Error serializing record: java.lang.UnsupportedOperationException: BSON Converter cannot convert java.util.ArrayList to BSON Document

Issue:-While adding a json data to the MongoDB collection, following error has been observed.MONGODB_13 - Error serializing record: java.lang.UnsupportedOperationException: BSON Converter cannot convert java.util.ArrayList to BSON DocumentSolution:- MongoDB atlas was reading the record as a list. As the JSON data was enclosed with the [].We have to set the JSON Content to JSON array of objects at the Origin side.

8 months ago

SindhuStreamSets Employee

posted in Share your Best Practices

Pipeline ID change.

Does the pipeline ID change, when any changes are made to the pipeline and re-published ?Pipeline ID is assigned during pipeline creation and does not change under any circumstances.

8 months ago

AkshayJadhavStreamSets Employee

posted in Share your Best Practices

Configuring Google Secret Manager Credential Store for data collector.

In this article, we will walk you through the necessary steps to configure and set up Google Secrets for use with Data Collector.Firstly, We have to create the secret on the GCP which we will be using in the pipeline. In this article, I am going to store the mysql username and password in secrets.These are the secret created in GCP:- The pre-requisite is to install the package Google Secret Manager Credential Store on the Data collector and then update the following properties in credential-stores.properties file.credentialStores=gcp credentialStore.gcp.config.project.id= <gcp-projectID> credentialStore.gcp.config.credentialsMode=jsonPath credentialStore.gcp.config.credentialsJsonPath=/Users/apple/.config/gcloud/application_default_credentials.jsonOnce done, restart the data collector. This implicit EL expression we are going to use in the pipeline credentials pipeline:Username => ${credential:get("gcp", "all@all", "akshay_sqlusername?latest")}Password => ${credential:get("

8 months ago

AkshayJadhavStreamSets Employee

posted in Share your Best Practices

SDC 5.7.X is not honouring the proxy configurations.

Environment:Streamsets Data Collector 5.7.X Proxy Configurations. Issue:There seems to be an issue with the latest version of the data collector (5.7.x). Specifically, it has come to our attention that the "http.nonProxyHosts" configuration is not being honored. Additionally, when executing the "ps -ef | grep sdc" command on the Linux host, the Java configurations are not visible. Solution:There is an issue with the data collector where proxy configurations are not working and will be fixed in further version(will keep updating the article once we have the exact fix version).The alternative/workaround is to just set it in the “Advanced Java Options” panel:-Dhttp.nonProxyHosts='*.azuresynapse.net'Any JVM configs specified directly will override those configurations and you could confirm by running “ps -ef | grep sdc” command on the Linux machine where data collector is running. NOTE: We have made some changes in latest version of data collector not to expose the proxy configuration hen

8 months ago

Sankar ArunachalamFan

posted in Share your Best Practices

Cloud DB migration - query

Can we configure multiple tables in same pipeline when we use JDBC Query consumer 1 as origin and JDBC producer 1 as destination. in my case i need to send 20 tables data to send from on-premise DB (db2) to cloud DB (Azure postgresql)?. i am thinking to avoid multiple pipelines creation in streamsets.

8 months ago

alejandro.alfonsoStreamSets Employee

asked in Share your Best Practices

Snowflake JDBC Driver v3.13.25 Behavior Change and SNOWFLAKE_11 Error

Error: If your Snowflake account name contains underscores and you're using Snowflake JDBC driver version 3.13.25 or higher, you might encounter the following error message:SNOWFLAKE_11 error. This error message indicates a communication problem with the message: "JDBC driver encountered a communication error. Message: Exception encountered for HTTP request: my-account.us-west-2.privatelink.snowflakecomputing.com: Name or service not known."Solution: To prevent this error, explicitly set allowUnderscoresInHost to 'true' in your JDBC connection string or properties. Refer to the Snowflake Community article for additional guidance.Affected Environments: This issue impacts environments meeting the following criteria:Snowflake account name contains underscores. JDBC is used for private link connections via the 'classic' regioned URL (e.g., https://my_account.us-west-2.privatelink.snowflakecomputing.com). Upgrading Snowflake JDBC driver from version 3.13.24 or lower to 3.13.25 or higher.

8 months ago

Arti_RichhariyaFan

asked in Share your Best Practices

Reading File from s3 and saving it as a same name

I am reading a file from s3 and after doing some manipulation in file i wanted to save it different location in s3 but with same name .how can i do this?please help.

9 months ago

Sankar ArunachalamFan

asked in Share your Best Practices

Cloud DB migration - query

9 months ago

LavenderStreamSets Employee

asked in Share your Best Practices

Pipeline Finisher shows error Unsatisfied precondition(s) '${record:eventType() == 'no-more-data'}'

Issue: Pipeline can run successfully but Pipeline Finisher stage shows error “Unsatisfied precondition(s) '${record:eventType() == 'no-more-data'}'”Resolution:This error is due to the configuration in Pipeline Finisher stage, On Record Error tab as Send to Error. Therefore, this error COMMON_0001 - Stage precondition: CONTAINER_0051 - Unsatisfied precondition(s) '${record:eventType() == 'no-more-data'}' is an expected behavior. In this case, the pipeline would still end when it receives no-more-data event, but it would keep generating these error records until no-more-data condition is met.To get rid of this error, you can change in On Record Error tab from Send to Error to Discard. You can read more about this configuration in our documentation: Stopping a Pipeline After Processing All Available Data.

9 months ago

alejandro.alfonsoStreamSets Employee

posted in Share your Best Practices

Legacy Control Hub & SAML - WEB_SSO_204 error

Problem:When attempting to log in to Legacy Control Hub with SAML enabled, users encounter an HTTP 401 error after trying to authenticate with the Identity Provider. The Login Audit trail provides the following information:HTTP Status Code: 401 Error Type: WEB_SSO_204 Response Status: urn:oasis:names:tc:SAML:2.0:status:RequesterSolution:The above error basically means that the Identity Provider has not granted authorization to the user, resulting in a 401 error response to Legacy Control Hub.This issue may stem from a misconfiguration of SAML settings between Legacy Control Hub and the Identity Provider. In most cases, the problem is related to the SAML keys used by Control Hub. Ensure that the public keys are valid and that the Identity Provider is utilizing the same keys. We recommend exchanging fresh copies of metadata and public keys between Legacy Control Hub and the Identity Provider. Afterward, clear your browser's cache and attempt to log in to Control Hub again. If you encount

9 months ago

AkshayJadhavStreamSets Employee

posted in Share your Best Practices

Error "DELTA_PATH_DOES_NOT_EXIST" while running Databricks Delta Lake pipeline.

Environment: Data Collector. Destination Databricks Delta Lake. Issue: The pipeline is encountering a "DELTA_PATH_DOES_NOT_EXIST" error, signaling that the specified path does not exist. To resolve this, we have manually created the file in Databricks.DELTA_LAKE_37 - Runnable process batch failed: DELTA_LAKE_12 - Could not perform SQL operation: java.sql.SQLException: [Databricks][DatabricksJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: org.apache.hive.service.cli.HiveSQLException: Error running query: [DELTA_PATH_DOES_NOT_EXIST] org.apache.spark.sql.AnalysisException: dbfs:/dash/DeltaLake/TEST doesn't exist at org.apache.spark.sql.hive.thriftserver.HiveThriftServerErrors$.runningQueryError(HiveThriftServerErrors.scala:48) at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.$anonfun$execute$1(SparkExecuteStatementOperation.scala:694) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at com.databricks.unit

9 months ago

AkshayJadhavStreamSets Employee

posted in Share your Best Practices

How to connect to Databricks and copy data into Delta Lake tables?

In this article, we will explore the process of creating a pipeline to transfer data from a MySQL database to a Delta Lake tables on Databricks.Environment:Data Collector on DataOps platform. Destination Databricks Delta Lake.These are the steps we need to follow to develop the pipeline: Login to Databricks UI » Compute » Create new compute then fill out all the details such as spark version, type of cluster, worker type. Once everything is finalised, click on create compute.Creating compute will take some sweet time.Now, we will have to fetch the following information from the cluster and populate into the pipeline configuration section. Token: In your Databricks workspace, click your Databricks username in the top bar, and then select User Settings from the dropdown » Develop » Token » Generate a new token. JDBC String: Click on the Databricks newly created cluster, scroll down and expand Advance Option tab » JDBC/ODBC. For staging purpose, we need to use AWS S3, Google Storage or

9 months ago

AkshayJadhavStreamSets Employee

asked in Share your Best Practices

Configuring Azure Key Vault Credentials Store for data collector.

In this article, we will walk you through the necessary steps to configure and set up Azure Secrets for use with Data Collector.Firstly, We have to create the secret on the Azure portal which we will be using in the pipeline. In this article, I am going to store the mysql username and password in secrets. The pre-requisite is to install the package Azure Key Vault Credentials Store on the Data collector and then update the following properties in credential-stores.properties file.credentialStores=azure credentialStore.azure.config.vault.url=https://<YOUR_KEY_VAULT>.vault.azure.net/ credentialStore.azure.config.client.id=<MUST BE SET> credentialStore.azure.config.client.key=<MUST BE SET>This is how you can create the secret and add the values to it:-Secret creating wizard. Added 2 secret.Once done, restart the data collector and add the following EL expression in the pipeline to fetch the username and password.Username => ${credential:get("azure", "all@all", "SQLus

9 months ago

AkshayJadhavStreamSets Employee

posted in Share your Best Practices

How to connect to Azure Storage SFTP with SFTP_Client Origin?

In this article, we will explore the process of accessing an Azure Storage Account's SFTP using a Data Collector configured with an SFTP/FTP/FTPS Client origin. Pre-requisite is to create the Azure SFTP account and for that you will need to perform the following steps:Login to Azure Portal → Storage Account → Create New Account → Make sure to enable the Hierarchical Namespace option to visible the SFTP option. Make sure to setup the Network connectivity section as per your environment. This will work like a firewall to your storage account which control the incoming connection to the storage account. Once the storage account is created navigate to the SFTP section and create the user to access the SFTP location. We also have to provide the container location where all the files will be saved or available. Once the user is created, you will be prompted the temporary autogenerated password. Now, Azure Storage account is created and SFTP service is enabled as well. Lets prepare the p

9 months ago

LavenderStreamSets Employee

posted in Share your Best Practices

Does SDC support IPv6?

Question: Does SDC support IPv6?Answer: Our Data Collector core language was done in Java and, consequently, supports IPv6. Since we’re on Java Stack, the IPv6 is mostly there for the core Data Collector pieces. For compatibility with Data Collector, it mainly depends on the source / target you want to connect whether it supports IPv6. Since SDC acts as a system integrator, for each stage, we would have to go through it and see if they are IPv6 compatible or not. For example: Oracle DB has started to support IPv6 connectivity since 12c, therefore theoretically Data Collector should also support it.Please note that there are some technologies (and versions) that don’t support IPv6. If the origin or destination doesn’t support it, we currently don't have plans to create workarounds or patches to allow it.Besides that, communications between Data Collector and Control Hub / Platform are done using IPv4, so it should be visible between the engine and the Platform. This is important if you

10 months ago

Anonymous

published in Share your Best Practices

Join the Write for Us programBlog

Ready to unleash your inner wordsmith? Join the Write for Us program. Your words. Our platform.Share your StreamSets use case and your technical expertise.The best part is you get paid for your contributions. Cha-ching! 💰 Inkspiration awaits.Sign up now

700

10 months ago

Page 1 / 3

Badge winners

vishwesh.margasahayamhas earned the badge Product expert
ajinkyahas earned the badge Innovator
Sanjeevhas earned the badge Eager to help
AkshayJadhavhas earned the badge Eager to help
john.durkinhas earned the badge Eager to help

Show all badges

Terms & Conditions

Sign up

Already have an account? Login

Social Login

Username *

E-mail address *

What I do... *

Data Leader Data Architect Data Engineer Data Scientist Other

Company *

Country *

Zip Code *

Marketing Communications

Yes No

Password *

I have read and Agree to the Website Terms of Service and I have read and acknowledged the Privacy Policy.

loginBox.register.email_repeat

Login to the community

No account yet? Create an account

Social Login

Username or Email

Password

Remember me

Forgot password?

Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.

Enter your e-mail address

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.