Skip to main content

StreamSets Platform

Get inspired and gain all the knowledge you need

1200 Topics
Sindhu
StreamSets Employee
SindhuStreamSets Employee
posted in Share your Best Practices
Pipeline ID change.

Does the pipeline ID change, when any changes are made to the pipeline and re-published ?Pipeline ID is assigned during pipeline creation and does not change under any circumstances.

340
Sindhu
StreamSets Employee
1 year ago
Francisco
Fan
FranciscoFan
asked in Community Articles and Got a Question?
HTTP Processor for POST request clearing header values

Hello! I’m using record header attributes as variables. For example, during pipeline execution, saving ${time:now()} in a header attribute with an Expression Evaluator, to be used in other stages. This is no problem for HTTP GET requests, as the data gets aggregated into existing records. The problem is with HTTP Processor for POST requests, where the response overwrites all existing data from before, body and headers.Is there anyway around this? I’m trying to use this header attribute value that I saved to be used in anohter HTTP POST request after the one I just mentioned, but the value is lost. Thanks in advance!

251
Bikram
Headliner
1 year ago
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Share your Best Practices
Configuring Google Secret Manager Credential Store for data collector.

In this article, we will walk you through the necessary steps to configure and set up Google Secrets for use with Data Collector.Firstly, We have to create the secret on the GCP which we will be using in the pipeline. In this article, I am going to store the mysql username and password in secrets.These are the secret created in GCP:- The pre-requisite is to install the package Google Secret Manager Credential Store on the Data collector and then update the following properties in credential-stores.properties file.credentialStores=gcp credentialStore.gcp.config.project.id= <gcp-projectID> credentialStore.gcp.config.credentialsMode=jsonPath credentialStore.gcp.config.credentialsJsonPath=/Users/apple/.config/gcloud/application_default_credentials.jsonOnce done, restart the data collector. This implicit EL expression we are going to use in the pipeline credentials pipeline:Username => ${credential:get("gcp", "all@all", "akshay_sqlusername?latest")}Password => ${credential:get("

1221
D
Opening Band
1 year ago
paul
StreamSets Employee
paulStreamSets Employee
posted in Community Articles and Got a Question?
SDC 5.7.x - Orchestration pipeline calling a Job Template does not function properly

Environment:StreamSets Data Collector (SDC) 5.7.0 & 5.7.1 An orchestration pipeline which uses Start Jobs stages The orchestration stage starts a job using a Job Template The orchestration stage is configured with the Run in Background setting disabled  Issue:In StreamSets Data Collector (SDC) 5.7.0 and 5.7.1, orchestration pipelines that use a Start Jobs stage to start a job using a Job Template do not function properly when Run in Background is disabled. With this configuration, the pipeline should wait for the job to complete before moving to the next stage, but instead the pipeline starts the job and immediately moves on, which can cause unexpected results for downstream jobs. Resolution:There is a fix planned for this issue in the SDC 5.8.0 release. In the short term, if your organization has an active support contact, you can contact StreamSets Support to obtain a patch to resolve the issue.  

300
paul
StreamSets Employee
1 year ago
AkshayJadhav
StreamSets Employee
AkshayJadhavStreamSets Employee
posted in Share your Best Practices
SDC 5.7.X is not honouring the proxy configurations.

Environment:Streamsets Data Collector 5.7.X Proxy Configurations. Issue:There seems to be an issue with the latest version of the data collector (5.7.x). Specifically, it has come to our attention that the "http.nonProxyHosts" configuration is not being honored. Additionally, when executing the "ps -ef | grep sdc" command on the Linux host, the Java configurations are not visible. Solution:There is an issue with the data collector where proxy configurations are not working and will be fixed in further version(will keep updating the article once we have the exact fix version).The alternative/workaround is to just set it in the “Advanced Java Options” panel:-Dhttp.nonProxyHosts='*.azuresynapse.net'Any JVM configs specified directly will override those configurations and you could confirm by running “ps -ef | grep sdc” command on the Linux machine where data collector is running.  NOTE: We have made some changes in latest version of data collector not to expose the proxy configuration hen

220
AkshayJadhav
StreamSets Employee
1 year ago
paul
StreamSets Employee
paulStreamSets Employee
posted in Community Articles and Got a Question?
SCH 3.x - Sharing Settings - Users/Group not showing up in the search box

IssueWhen updating the sharing settings for pipelines, jobs, etc. typing in a user or group that is known to exist in the organization doesn’t show up in the search box.ResolutionThis occurs when there are more than 50 users and/or groups. Control Hub preloads 50 users and groups for the purposes of autocomplete. If a user or group doesn’t show up, you can type in the full username (user@org) or group name (group@org) and click the add button, and it will work.NoteThis issue is specific to legacy StreamSets Control Hub 3.x. In Platform, the search box has been improved and should autocomplete for all users and groups. 

90
paul
StreamSets Employee
1 year ago
blakeorsak
Fan
blakeorsakFan
posted in Community Articles and Got a Question?
Azure_02 Error when creating an Azure Connection in Control Hub

The Issue:  When attempting to create a Connection for an Azure account, you receive the following error message: ADLS_02 - Failed to connect to Azure Data Lake Store: ‘Operation failed: “An HTTP header that’s mandatory for this request is not specified”,400,PUT, <URL>The Solution:  This error could potentially indicate that the Connection type set in Control Hub and the Azure account type do not match. When using a Data Collector version that is higher than 5.4, you are presented with a number of options when creating a new connection to Azure. These options include: Azure Blob Storage Azure Data Lake Storage Gen2 Azure Synapse When this error is encountered, before troubleshooting further it is important to first check that the connection type that was chosen here matches the Azure Account type you are connecting to. This can be verified by checking the Azure Account FQDN. For example: An FQDN like example.dfs.core.windows.net would be used for an Azure Data Lake Storage Gen2 c

370
blakeorsak
Fan
1 year ago
catchcharan
Fan
catchcharanFan
asked in Community Articles and Got a Question?
Certification - Proctor U code

Hi Team, I have paid for white belt certification and received a receipt number as well with payment confirmation for white belt exam.I never received any email from ProctorU which consists of code that I need to enter to proceed to schedule an exam. It’s been 15 days now.Can anyone help on how to proceed with this issue? Thanks,

351
john.durkin
StreamSets Employee
1 year ago
N
Fan
NateFan
asked in Community Articles and Got a Question?
Platform SDK for Python Version 5.2.1 pipeline.committer return

I am using the python SDK and I when I use the method “.committer” or “.last_modified_by” on a pipeline object, I am expecting to get back an email that I am able to see under “Committed By” on the platform. However, I get back a long string that I can’t make out:sudo code of what I am doing: sch = ControlHub(credential_id="cred_id,token = token)print(sch.pipelines.get(commit_id = commit_id).last_modified_by) and instead of getting a string of an email returned, I am getting something like this returned:a83b3195-61f4-11ee-a6e4-…..@b5f95dcd-5f00-11ec-b405-…… Would you happen to know how I can turn this string into a valid email? I couldn’t find anything on this topic in the docs.  

251
Mihir Kale
Roadie
1 year ago
john.mcavoy
StreamSets Employee
john.mcavoyStreamSets Employee
asked in Community Articles and Got a Question?
Kafka Stage fails with error "Request joining group due to: rebalance failed due to 'null' (DisconnectException)"

IssueA Kafka stage fails to connect to the broker due to the follow error stack trace:INFO ConsumerCoordinator - (Re-)joining groupINFO NetworkClient - Disconnecting from node xxx due to socket connection setup timeout. The timeout value is 8424 ms.INFO ConsumerCoordinator - Group coordinator is unavailable or invalid due to cause: null. isDisconnected: true. Rediscovery will be attempted.2023-09-27 13:07:58,776 INFO ConsumerCoordinator - Request joining group due to: rebalance failed due to 'null' (DisconnectException)BackgroundThis specific Kafka Client exception appears when the Kafka Client’s socket-connection-setup-timeout-max-ms is set too low, causing it to attempt to re-join the group.The maximum amount of time the client will wait for the socket connection to be established. The connection setup timeout will increase exponentially for each consecutive connection failure up to this maximum. SolutionWhen this property is set too low, the Kafka Client will time out during the

23660
john.mcavoy
StreamSets Employee
1 year ago
S
Fan
Sankar ArunachalamFan
posted in Share your Best Practices
Cloud DB migration - query

Can we configure multiple tables in same pipeline when we use JDBC Query consumer 1 as origin and JDBC producer 1 as destination. in my case i need to send 20 tables data to send from on-premise DB (db2) to cloud DB (Azure postgresql)?. i am thinking to avoid multiple pipelines creation in streamsets.

101
Pradeep
StreamSets Employee
1 year ago
S
Roadie
shivrajRoadie
asked in StreamSets Academy
Facing an error while testing a pipeline on STF 5.x

We are facing some connection issue from STF side while executing a test code, that might be regarding the organization ID not sure why this issue is occurring it saying for organization the url STF  calling is “api/security/v3/orgs/e4cbaa88-611a-11ed-b310-c96332d26f9a” but url must be ”http://api/security/v3/orgs/e4cbaa88-611a-11ed-b310-c96332d26f9a?”. I’m not sure how to fix this Error. E           requests.exceptions.MissingSchema: Invalid URL 

211
S
Roadie
1 year ago
alejandro.alfonso
StreamSets Employee
alejandro.alfonsoStreamSets Employee
asked in Share your Best Practices
Snowflake JDBC Driver v3.13.25 Behavior Change and SNOWFLAKE_11 Error

Error: If your Snowflake account name contains underscores and you're using Snowflake JDBC driver version 3.13.25 or higher, you might encounter the following error message:SNOWFLAKE_11 error. This error message indicates a communication problem with the message: "JDBC driver encountered a communication error. Message: Exception encountered for HTTP request: my-account.us-west-2.privatelink.snowflakecomputing.com: Name or service not known."Solution: To prevent this error, explicitly set allowUnderscoresInHost to 'true' in your JDBC connection string or properties. Refer to the Snowflake Community article for additional guidance.Affected Environments: This issue impacts environments meeting the following criteria:Snowflake account name contains underscores. JDBC is used for private link connections via the 'classic' regioned URL (e.g., https://my_account.us-west-2.privatelink.snowflakecomputing.com). Upgrading Snowflake JDBC driver from version 3.13.24 or lower to 3.13.25 or higher.

1980
alejandro.alfonso
StreamSets Employee
1 year ago
A
Fan
Arti_RichhariyaFan
asked in Share your Best Practices
Reading File from s3 and saving it as a same name

I am reading a file from s3 and after doing some manipulation in file i wanted to save it different location in s3 but with same name .how can i do this?please help. 

321
Bikram
Headliner
1 year ago
paul
StreamSets Employee
paulStreamSets Employee
posted in Community Articles and Got a Question?
StreamSets Control Hub rpm package for Red Hat Enterprise Linux 8

QuestionThere are rpm packages for Red Hat 6 and 7, but not for Red Hat 8? AnswerStreamSets has tested the RHEL 7 package on 8 and certified it for use. There are no plans to release a separate rpm package specifically labeled for RHEL 8.

330
paul
StreamSets Employee
1 year ago
P
Fan
Priya151997Fan
asked in Community Articles and Got a Question?
After S3 Event need to capture file name in expression evaluator

Hi All,we are new in streamsets !!In our usecase we are uploading file in s3 which predefined name as abcd_sggshs_31-04-2018.csvwe need to capture this above parameter which are in our file name,we are generating events after uploading file in s3,using expression evaluator we are trying to capture this parameter in json file in s3.but we are not getting desired output.below are the expression we are using please suggest the correct expression to get the file name.   

2778
R
StreamSets Employee
1 year ago
som
Fan
somFan
asked in Community Articles and Got a Question?
Jobs do not carry over offset value when migrate from one server to another.

The offset is always stored in the Control Hub (SCH). The Data Collector (SDC) stores the offset to the offset.json file and sends the information to the SCH periodically. When 'Enable Failover' is checked, the offset is migratable i.e the SCH can start a job on a different SDC (SDC_2) with the last-saved offset (from the previous SDC - SDC_1, which is not available anymore). If 'Enable Failover' is not checked, the offset is tied to a specific SDC ID and it does not migrate i.e the SCH cannot start a job with the last-saved offset from another SDC (SDC_1) on a different SDC (SDC_2). On a new SDC, the job will run the pipeline from the beginning. Failover needs to be enabled for the offset to be migrated.

350
som
Fan
1 year ago
K
Fan
kushal_upadhyayFan
asked in Community Articles and Got a Question?
Reading avro file from a bucket similar to S3 and trying to convert into parquet

I am reading an avro file from a custom cloud similar to S3 and trying to conver it into parquet file using whole file evaluator but its giving an error  Record1-Error Record1 CONVERT_01 - Failed to validate record is a whole file data format : java.lang.IllegalArgumentException: Record does not contain the mandatory fields /fileRef,/fileInfo,/fileInfo/size for Whole File Format. : (View Stack Trace...   

2083
gccoulter
StreamSets Employee
1 year ago
gccoulter
StreamSets Employee
gccoulterStreamSets Employee
posted in Community Articles and Got a Question?
Converting an AVRO file (with Schema) to Parquet, using Whole File Transformer

Here we will be covering how to convert an AVRO file with Schema, to a Parquet file.STEP 1Set your pipeline up as follows;Directory - Whole File Transformer - Local FS STEP 2Configure your Directory Origin as follows; GENERAL TAB FILES TAB Please ensure you make the amendments necessary to certain fields to suit your own set up, e.g. File Name Pattern. Keep the rest as in the screenshot. POST PROCESSING TAB DATA FORMAT TAB STEP 3Configure Whole File Transformer stage as follows; GENERAL TAB JOB TAB*amend file directory as necessary AVRO TO PARQUET TAB If you preview the pipeline, ensure to select the following boxes, which will allow you to click in to each record and schema;  You should now have a .parquet file from your original AVRO.

1120
gccoulter
StreamSets Employee
1 year ago
S
Roadie
skyRoadie
asked in Community Articles and Got a Question?
Produce Events Error

When I am testing the Produce Events of a component. When I uncheck the "Produce Events" box, the component configuration displays normally. However, when I check it, I always encounter the following error. Do you know what might be causing this? Has anyone else experienced this issue?  

364
Bikram
Headliner
1 year ago
nathan
StreamSets Employee
nathanStreamSets Employee
posted in Community Articles and Got a Question?
Technical Service Bulletin 2023-08-25 (TSB) - Azure Synapse, Delta Lake, Google BigQuery, and Snowflake destinations issues with DateTime data

StreamSets Support team would like to inform you of an issue where StreamSets Data Collector pipelines using destinations such as Azure Synapse, Delta Lake, Google BigQuery, or Snowflake might corrupt the data for the DATE/DATETIME/TIME fields.Products Affected: StreamSets Data Collector Releases Affected:  StreamSets Data Collector 5.6.0 and prior versions Users Affected: StreamSets Data Collector pipelines using Azure Synapse destination, Delta Lake destination, Google BigQuery destination, and Snowflake destination. Severity: High Description: It has been identified that in some cases, DATE/DATETIME/TIME datatype fields could be corrupted. Users could be finding that their DATE/DATETIME/TIME data fields contain incorrect dates. This issue happens only if all the following conditions take place:The pipeline writes to one of the following stages: Azure Synapse destination Delta Lake destination Google BigQuery destination Snowflake destination.  The pipeline writes to multiple tab

140
nathan
StreamSets Employee
1 year ago
nathan
StreamSets Employee
nathanStreamSets Employee
posted in Community Articles and Got a Question?
Technical Service Bulletin 2023-07 (TSB) - Changes in Amazon S3 Stage Properties  from StreamSets Data Collector version 5.6.0

StreamSets Support Team would like to inform you that starting with Data Collector 5.6.0 pipelines with Amazon S3 stages fail to execute if the "Bucket" or "Bucket and path" property contains a forward slash (/). Products Affected: StreamSets Data Collector Releases Affected: Starting from StreamSets Data Collector version 5.6.0 Users Affected: StreamSets Data Collector pipelines that include the forward slash (/) in the bucket property of Amazon S3 stages. Severity: Medium/High Description: Starting with version 5.6.0, you can no longer include the forward slash (/) in the following properties for Amazon S3 stages due to an Amazon Web Services SDK upgrade:Bucket property for the Amazon S3 origin Bucket and path property for the Amazon S3 destination and executorAdditional details regarding this change can be found in the Upgrade Impact documentation.Immediate action required: YesResolution: Avoid using the forward slash (/) in the bucket name as this may cause signature mismatch error

40
nathan
StreamSets Employee
1 year ago
nathan
StreamSets Employee
nathanStreamSets Employee
asked in Community Articles and Got a Question?
Technical Service Bulletin 2023-09-01 (TSB)- Thread Leak In Snowflake Stages

StreamSets Support Team would like to inform you of an issue where StreamSets Data Collector pipelines using Snowflake stages might have problems with threads not being released.  Products Affected: StreamSets Data Collector Releases Affected: StreamSets Data Collector version 5.6.0 and 5.6.1 Users Affected: StreamSets Data Collector pipelines using Snowflake stages. Severity: High Description: A minor thread leak has been detected in all the Snowflake stages. If a user environment involves a substantial number of Snowflake pipelines, users may observe this thread leak behavior, which can impact the overall system performance and resource utilization.Immediate action required: Yes, upgrade to SDC 5.6.3 or 5.7.X.Workaround:  NoneResolution: Fix for this issue has been included in SDC 5.6.3 and 5.7.x release. For your questions, please contact our Support Team via our Support portal or at support@streamsets.com.

250
nathan
StreamSets Employee
1 year ago
nicholas.mendoza
Fan
nicholas.mendozaFan
posted in Community Articles and Got a Question?
Underscore in account name can cause SNOWFLAKE_11 error

When trying to connect to Snowflake in SDC, you may encounter the following error:SNOWFLAKE_11 - Could not create SQL DataSource: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: JDBC driver encountered communication error. Message: Exception encountered for HTTP request: Remote host terminated the handshake.This error may be caused by some configuration mistakes in the stage. One possible cause is underscore(s) in the account name. The Snowflake JDBC driver does not support the use of underscores in URLs, which include the account name. If your account name does contain underscores, and you are receiving a similar error, a workaround would be to use a hyphen instead.

500
nicholas.mendoza
Fan
1 year ago
himanshu1234567
Discovered Fame
himanshu1234567Discovered Fame
asked in Community Articles and Got a Question?
ImportError: cannot import name 'JSONDecodeError' from 'requests.exceptions'

while running python program using streamsets python sdk we are getting error File "/var/lib/jenkins/.local/lib/python3.8/site-packages/streamsets/sdk/exceptions.py", line 5, in <module> from requests.exceptions import HTTPError, JSONDecodeError ImportError: cannot import name 'JSONDecodeError' from 'requests.exceptions'we have found solution for this but we want to know why we are getting this error.Solution for this is manually we are replacing line “from requests.exceptions import HTTPError, JSONDecodeError”   with “from json import JSONDecodeError”

32801
S
StreamSets Employee
1 year ago

Badge winners

Show all badges
Terms & Conditions