Skip to main content

StreamSets Platform

Get inspired and gain all the knowledge you need

1200 Topics
S
Fan
Sankar ArunachalamFan
asked in Share your Best Practices
Cloud DB migration - query

Can we configure multiple tables in same pipeline when we use JDBC Query consumer 1 as origin and JDBC producer 1 as destination. in my case i need to send 20 tables data to send from on-premise DB (db2) to cloud DB (Azure postgresql)?. i am thinking to avoid multiple pipelines creation in streamsets.

161
Bikram
Headliner
1 year ago
Lavender
StreamSets Employee
LavenderStreamSets Employee
asked in Share your Best Practices
Pipeline Finisher shows error Unsatisfied precondition(s) '${record:eventType() == 'no-more-data'}'

Issue: Pipeline can run successfully but Pipeline Finisher stage shows error “Unsatisfied precondition(s) '${record:eventType() == 'no-more-data'}'”Resolution:This error is due to the configuration in Pipeline Finisher stage, On Record Error tab as Send to Error. Therefore, this error COMMON_0001 - Stage precondition: CONTAINER_0051 - Unsatisfied precondition(s) '${record:eventType() == 'no-more-data'}' is an expected behavior. In this case, the pipeline would still end when it receives no-more-data event, but it would keep generating these error records until no-more-data condition is met.To get rid of this error, you can change in On Record Error tab from Send to Error to Discard. You can read more about this configuration in our documentation: Stopping a Pipeline After Processing All Available Data. 

1050
Lavender
StreamSets Employee
1 year ago
alejandro.alfonso
StreamSets Employee
alejandro.alfonsoStreamSets Employee
posted in Share your Best Practices
Legacy Control Hub & SAML - WEB_SSO_204 error

Problem:When attempting to log in to Legacy Control Hub with SAML enabled, users encounter an HTTP 401 error after trying to authenticate with the Identity Provider. The Login Audit trail provides the following information:HTTP Status Code: 401 Error Type: WEB_SSO_204 Response Status: urn:oasis:names:tc:SAML:2.0:status:RequesterSolution:The above error basically means that the Identity Provider has not granted authorization to the user, resulting in a 401 error response to Legacy Control Hub.This issue may stem from a misconfiguration of SAML settings between Legacy Control Hub and the Identity Provider. In most cases, the problem is related to the SAML keys used by Control Hub. Ensure that the public keys are valid and that the Identity Provider is utilizing the same keys. We recommend exchanging fresh copies of metadata and public keys between Legacy Control Hub and the Identity Provider. Afterward, clear your browser's cache and attempt to log in to Control Hub again. If you encount

80
alejandro.alfonso
StreamSets Employee
1 year ago
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Share your Best Practices
Error "DELTA_PATH_DOES_NOT_EXIST" while running Databricks Delta Lake pipeline.

Environment:  Data Collector. Destination Databricks Delta Lake. Issue:  The pipeline is encountering a "DELTA_PATH_DOES_NOT_EXIST" error, signaling that the specified path does not exist. To resolve this, we have manually created the file in Databricks.DELTA_LAKE_37 - Runnable process batch failed: DELTA_LAKE_12 - Could not perform SQL operation: java.sql.SQLException: [Databricks][DatabricksJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: org.apache.hive.service.cli.HiveSQLException: Error running query: [DELTA_PATH_DOES_NOT_EXIST] org.apache.spark.sql.AnalysisException: dbfs:/dash/DeltaLake/TEST doesn't exist at org.apache.spark.sql.hive.thriftserver.HiveThriftServerErrors$.runningQueryError(HiveThriftServerErrors.scala:48) at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.$anonfun$execute$1(SparkExecuteStatementOperation.scala:694) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at com.databricks.unit

4130
AkshayJadhav
StreamSets Employee
1 year ago
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Share your Best Practices
How to connect to Databricks and copy data into Delta Lake tables?

In this article, we will explore the process of creating a pipeline to transfer data from a MySQL database to a Delta Lake tables on Databricks.Environment:Data Collector on DataOps platform. Destination Databricks Delta Lake.These are the steps we need to follow to develop the pipeline: Login to Databricks UI » Compute » Create new compute then fill out all the details such as spark version, type of cluster, worker type. Once everything is finalised, click on create compute.Creating compute will take some sweet time.Now, we will have to fetch the following information from the cluster and populate into the pipeline configuration section. Token: In your Databricks workspace, click your Databricks username in the top bar, and then select User Settings from the dropdown » Develop » Token » Generate a new token. JDBC String: Click on the Databricks newly created cluster, scroll down and expand Advance Option tab » JDBC/ODBC. For staging purpose, we need to use AWS S3, Google Storage or

280
AkshayJadhav
StreamSets Employee
1 year ago
paul
StreamSets Employee
paulStreamSets Employee
posted in Community Articles and Got a Question?
SCH 3.x On-Prem - Database upgrade and potential subsequent rollback

Upgrading one or both of the backend databases for SCH 3.x (relational database and timeseries database) has no effect on SCH itself assuming that you are upgrading to a supported version of the database. You will need to either bring down SCH during the upgrade, or put the database into read-only mode, and of course take a backup of the database before beginning the upgrade.In a scenario where a subsequent rollback of the database to the backed-up version is contemplated, be aware that there will be some data loss, even if there were no new jobs created or other changes made (code freeze). Some of this data includes, but is not limited to:Any clean up of the DB Writing to InfluxDB Writing to the PostgreSQL Timseries DB (job metrics that don’t go to InfluxDB but to the SQL DB) Updating jobs statuses Running scheduler tasksThese data would need to be manually reconstructed from the upgraded database and backported to the rolled-back database.If you have any additional questions, please

110
paul
StreamSets Employee
1 year ago
paul
StreamSets Employee
paulStreamSets Employee
posted in Community Articles and Got a Question?
CONTROL_HUB_API_00 error in SDC log

In SDC versions up to 5.5.x, you may see CONTROL_HUB_API_00 errors which can cause pipelines and jobs to fail. There is no context in the logs as to what was happening that led up the to the error; it’s just a generic error that occurs when any type of API call to SCH fails.In SDC 5.6.x, this error was removed and more helpful logging was added that captures the specific API call that was being made and what the exception was. If you are encountering the CONTROL_HUB_API_00 error in an older version of SDC, StreamSets Support recommends either upgrading to at least 5.6.x, or install SDC 5.6.x or higher in a non-production environment and reproduce the issue. This will allow StreamSets Support to more effectively assist you and find the root cause of the issue.

70
paul
StreamSets Employee
1 year ago
john.mcavoy
StreamSets Employee
john.mcavoyStreamSets Employee
asked in Community Articles and Got a Question?
MapR Client Library Upgrade causes "Provider could not be instantiated" or "ClassNotFoundException"

IssueAfter upgrading MapR client libraries or applying a MapR client patch on an SDC node, pipelines using Hadoop stages fail with some Provider could not be instantiated error or ClassNotFoundException at startup, indicating some Hadoop class cannot be found within the Java classpath.BackgroundDue to licensing restrictions, StreamSets cannot distribute MapR libraries with Data Collector. As a result, you must perform additional steps to enable the Data Collector machine to connect to MapR.When the bin/streamsets setup-mapr prerequisite is completed, SDC creates symlinks to the installed MapR Client libraries on the system in the streamsets-libs/streamsets-datacollector-mapr-version-lib/lib directory.When applying a MapR Client patch or upgrading the MapR Client package on the SDC machine, some JAR files in the MapR client installation patch will be renamed which will cause broken symlinks within SDC’s MapR stage library directory.SolutionThe Provider could not be instantiated exceptio

110
john.mcavoy
StreamSets Employee
1 year ago
bsayed
Roadie
bsayedRoadie
asked in Community Articles and Got a Question?
Field mapper not mapping all fields

I have a field mapper to remap camel case fields to snake case. This is the message i am trying to save:{   "id": "e35fd529-613f-5e79-036f-f37c2801041d",   "batchName": "SIT Axle3+ 9_12_test",   "batchSize": 35,   "numberOfAudited": 35,   "batchType": "TRIP_BASED",   "batchMode": "RANDOM",   "batchFacility": "71",   "batchDirection": "West",   "batchPlaza": "",   "batchLane": "",   "fromTimestamp": 1694538060,   "toTimestamp": 1694566800,   "createdTimestamp": 1694626323,   "createdBy": "mmccammon@etcc.com",   "completedTimestamp": 1697039389,   "completedBy": "asayed@etcc.com",   "auditDetailList": [      {         "id": "a4f274d5-8de5-4c54-ae70-b6a498ab4e9e",         "auditBatchId": "e35fd529-613f-5e79-036f-f37c2801041d",         "eventId": "49dcbe2c-51a8-11ee-9348-3cecefd01508",         "eventType": "TRIP",         "eventTimeStamp": 1694549374,         "auditResult": false,         "auditCorrections": [],         "auditedTimestamp": 1694635817,         "auditedBy": "ppradeep@etcc.co

14819
bsayed
Roadie
1 year ago
A
Anonymous
published in Community Articles and Got a Question?
Zero to StreamSetsTutorial

An in-depth guide into starting StreamSets, deploying engines, and running pipelines. Overall architecture recapThe control plane of StreamSets Platform, Control Hub, hosted in the cloud. The data plane encompasses StreamSets Data Collectors (SDCs), which should be installed in a customer’s network, be it in the cloud or on-premise.  Consider these options for SDC installation (not all of them are recommended for production scenario):  Tarball (binary download) Docker container Windows PC not supported   OK – the fastest to implement Linux/Mac OK with configured prerequisites  Depending on your choice, find below instructions for major steps of tarball and Docker options. Provide and prepare your machineUse your own laptop or create a VM in the cloud of your choice. VM size should be determined by the minimum requirements for SDC installation: 2 vCores, 1GB RAM, 6GB disk space.If on Linux, run a system update command, like yum updat

1210
A
1 year ago
V
Fan
vivek.sharmaFan
asked in Community Articles and Got a Question?
Json to XML with namespaces

hello,I’m new to streamsets so it might be an obvious question!I’m using “data generator” to accept the json input and transform to XML, however I want to add few namespaces in the generated XML output. Please can anyone suggest how to achieve that?This is the output “data generator” is producing:<?xml version="1.0" encoding="UTF-8" standalone="no"?><root> <id>file</id> <value>File</value> <popup> <menuitem> <value>New</value> <onclick>CreateDoc()</onclick> </menuitem> <menuitem> <value>Open</value> <onclick>OpenDoc()</onclick> </menuitem> <menuitem> <value>Save</value> <onclick>SaveDoc()</onclick> </menuitem> </popup></root>What changes do I need to make it something like this (the element id is under “b” namespace):<?xml version="1.0" encoding="UTF-8" standalone="no" ?&g

380
V
Fan
1 year ago
L
Fan
levanyeFan
asked in Community Articles and Got a Question?
running error pipeline couldn`t force stop

 Running Error pipeline couldn`t stop, when I click the button of force stop, nothing happened, what should i do?

6279
saleempothiwala
Headliner
1 year ago
U
Fan
uday8770Fan
asked in Community Articles and Got a Question?
Metadata driven ingestion framework

HiI'm currently working on implementing a metadata-driven pipeline. One of the key components of this pipeline is a job that's responsible for transferring data from Teradata to Snowflake. To make this process dynamic and reusable, I've introduced a job which will load the data to snowflake.Furthermore, I've established a separate pipeline that serves the purpose of fetching metadata from Teradata. This metadata contains important information, including the values for ${dbname} and ${Tname}. The intention is to pass these parameter values to the "startjobs" process. However, I'm currently unsure about how these parameter values will be effectively utilized within the job I've created for the data transfer from Teradata to Snowflake. I would greatly appreciate any guidance or assistance in this regard.  

1165
saleempothiwala
Headliner
1 year ago
Paawan Angra
Fan
Paawan AngraFan
asked in Community Articles and Got a Question?
Start Jobs origin throws error regarding runtime parameters

I am trying to create a job instance from the job template. I have passed the runtime parameters to the start job component but get the following error. - Start failed for job template ID: START_JOB_14 - Start failed for job template ID: abc: com.streamsets.pipeline.lib.ControlHubApiException: Unexpected exception from Control Hub: START_JOB_05 - Failed to parse runtime parameters for job ID: abc, error: 400: I have made sure that the parameter name is correct. Please help me with why is it not able to parse the runtime parameters?  

668
saleempothiwala
Headliner
1 year ago
cortiez21
Fan
cortiez21Fan
asked in Community Articles and Got a Question?
Common IDP

Hello, AVG has quarantined CefSharp.BrowserSubprocess.exe as a possible IDP—generic infection. Slido doesn't function, and it's been quarantined.How should I proceed? 

121
Bikram
Headliner
1 year ago
S
Roadie
skyRoadie
asked in Community Articles and Got a Question?
Encrypt and Decrypt Fields 1 error

When I use the Encrypt and Decrypt component to encrypt a string field, I encounter the following error. Could you please tell me what might be causing this issue? Also, when I convert the string type to a byte array or base64 characters, it doesn't work either.

231
Bikram
Headliner
1 year ago
S
Fan
Satish PanchwareFan
asked in Community Articles and Got a Question?
Facing issues while decrypting text

I am working a sample involving encryption and decryption and facing issues.Here is the problem:While using the "Encrypt and Decrypt Fields" processor when we encrypt something using it, it outputs the resulting fields in a ByteArray format, and when we save it somewhere like localFS the ByteArray gets converted into String Format,Now when it comes to the decryption part, The processor needs ByteArray format only , So when I try to convert the String into Byte array it results in some unrecognizable characters which causes error.Encryption:Decryption:Please let me know if you have any pointers.

856
Bikram
Headliner
1 year ago
S
Roadie
shivrajRoadie
asked in StreamSets Academy
STF

In STF 3.x we have used "sch_authoring_sdc.get_history(pipeline)"  to get all the information, for 4.x do we have anything like that for accessing the history? as get_history is not working with STF 4.x?

452
S
Roadie
1 year ago
L
Fan
lizlarssonFan
asked in Community Articles and Got a Question?
Discarding records but passing value to next stage

Hi!We have a pipeline where we need to use two http stages. First we need to get all objects of type A and then for each A object we need to get all objects of type B.1. First we have an http origin where we get all objects of type A. The only thing of interest in the response is the id of object A2. For each A object, we make another http request (using a http client stage) to retrieve all the objects of type B. The url looks like this:https://www.example.com/rest/1.0/foo/${record:value('/id')}/bari.e. we use the id of A which we got in the first stage.   We are not really interested in the records we get in stage one. The only thing we are interested in is the id which will be used in the URL in stage two.How can we discard all records received in stage one and just pass the id to stage two? Is it possible to write to an environment variable?Any help would be appreciated!

353
Bikram
Headliner
1 year ago
R
Fan
rajasriramFan
asked in StreamSets Academy
Read file from ADLS Origin and Update same file in ADLS destination stage

I have a ADLS origin stage where I have a data file. The data file needs to be updated with a different delimiter value in it. After updating the values in the file, I need to overwrite it back to the same file. I am using jython evaluator to do the updates. However, I am getting a new file created in ADLS instead of writing to the existing file.Here is my Jython code that I am using   try: text_attribute = record.value["text"] unicode_text = unicode(text_attribute,'utf-8') modified_text = unicode_text.replace(u'|', u'~') record.value["text"]=modified_text sdc.output.write(record) except Exception as e: sdc.error.write(record, str(e))My Pipeline :ADLS Origin Stage -→ Jython Evaluator -→ ADLS Destination Stage

70
R
Fan
1 year ago
S
Fan
sudarshan_shinde1Fan
asked in Community Articles and Got a Question?
Get list of pipeline name using Python SDK

Hi Team,I am using below mentioned method to get list of all pipelines.But it is returning all the values like version, commit id.I want only pipeline name.sch = ControlHub(credential_id, token)pipeline_list = sch.pipelines.get_all() I tried sch.pipeline.name but it is not supporting.

864
Kirti
StreamSets Employee
1 year ago
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
asked in Share your Best Practices
Configuring Azure Key Vault Credentials Store for data collector.

In this article, we will walk you through the necessary steps to configure and set up Azure Secrets for use with Data Collector.Firstly, We have to create the secret on the Azure portal which we will be using in the pipeline. In this article, I am going to store the mysql username and password in secrets.  The pre-requisite is to install the package Azure Key Vault Credentials Store on the Data collector and then update the following properties in credential-stores.properties file.credentialStores=azure credentialStore.azure.config.vault.url=https://<YOUR_KEY_VAULT>.vault.azure.net/ credentialStore.azure.config.client.id=<MUST BE SET> credentialStore.azure.config.client.key=<MUST BE SET>This is how you can create the secret and add the values to it:-Secret creating wizard. Added 2 secret.Once done, restart the data collector and add the following EL expression in the pipeline to fetch the username and password.Username => ${credential:get("azure", "all@all", "SQLus

760
AkshayJadhav
StreamSets Employee
1 year ago
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Share your Best Practices
How to connect to Azure Storage SFTP with SFTP_Client Origin?

In this article, we will explore the process of accessing an Azure Storage Account's SFTP using a Data Collector configured with an SFTP/FTP/FTPS Client origin. Pre-requisite is to create the Azure SFTP account and for that you will need to perform the following steps:Login to Azure Portal → Storage Account → Create New Account →  Make sure to enable the Hierarchical Namespace option to visible the SFTP option. Make sure to setup the Network connectivity section as per your environment. This will work like a firewall to your storage account which control the incoming connection to the storage account. Once the storage account is created navigate to the SFTP section and create the user to access the SFTP location. We also have to provide the container location where all the files will be saved or available. Once the user is created, you will be prompted the temporary autogenerated password.  Now, Azure Storage account is created and SFTP service is enabled as well. Lets prepare the p

430
AkshayJadhav
StreamSets Employee
1 year ago
J
Fan
Jhon01Fan
asked in Community Articles and Got a Question?
The API functions correctly in Speakatoo POSTMAN but is not working within my integrated application.

"I've noticed that the Speakatoo API functions properly when tested in POSTMAN, but it's not performing as expected within my integrated application. I'm seeking assistance to understand and address the issue with the API integration in my application."

271
Bikram
Headliner
1 year ago
abibatoul20
Fan
abibatoul20Fan
asked in Community Articles and Got a Question?
Enable StreamSets Control Hub

Bonsoir, j’ai installé StreamSets Data Collector mais je veux activer le control hub. Quand j’essaie ça me génère l’erreur suivante : DPM Login failed, status code '403': {"message":"Authentication failed"} ( View Details... )

331
Kirti
StreamSets Employee
1 year ago

Badge winners

Show all badges
Terms & Conditions