Community Articles and Got a Question?

Can't find what you're looking for? Ask it here or check out the Community articles

  • 945 Topics
  • 2,248 Replies

945 Topics

S
Fan
sudarshan_shinde1Fan
asked in Community Articles and Got a Question?

Export external jar using python sdk.

Hi,We need to export jar files/libraries of specific engine of one control hub to another control hub using Python SDK. How can we achieve it? Thanks.

0
S
Fan
8 hours ago
simba01
Fan
simba01Fan
asked in Community Articles and Got a Question?

Why does the Speakatoo API function properly in POSTMAN but not within my integration?

The Speakatoo API functions correctly in POSTMAN, but it isn't working as expected in my integration for Text To Speech conversion. What might be the issue?

0
simba01
Fan
1 day ago
M
Fan
meowwFan
asked in Community Articles and Got a Question?

S3 pipelines on SDC v5.6.0 failing with AmazonS3Exception:SignatureDoesNotMatch after upgrading to v5.6.0 from v5.5.0

we upgraded data collector to v5.6.0 from v5.5.0 recently… but since then getting our s3 pipelines are failing with below ERROR.. we are using `Instance Profile` as authentication method..Also data collector with v5.5.0 which are deployed same (AWS acc)environment are working fines with s3 pipelines & instance_profile authentication method. anyone else faced similar kind of problem?RETRY: S3_21 - Unable to write object to Amazon S3, reason : com.amazonaws.services.s3.model.AmazonS3Exception: The request signature we calculated does not match the signature you provided. Check your key and signing method. (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: 3VFHG02TG4GR3KTR; S3 Extended Request ID: PePeTM0p51ZFcECJ/Mw6PU9nK4Km6AWNEaWhcykqj2ovDW/IbGcMtcaK+xcRxlMoPuRlCyxh2vo=; Proxy: null), S3 Extended Request ID: PePeTM0p51ZFcECJ/Mw6PU9nK4Km6AWNEaWhcykqj2ovDW/IbGcMtcaK+xcRxlMoPuRlCyxh2vo=

1
Lavender
StreamSets Employee
1 day ago
lakshman
Fan
lakshmanFan
asked in Community Articles and Got a Question?

Reading CSV file it shows extra record .

 My csv file has only Two Record But it shows ,extra records in stream sets Environment.this is my csv record data  but Building Information column had a paragraph data.In stream sets 5 records Showing .anyone can suggest how to handle this file. ​​​​​

3
Bikram
Headliner
3 days ago
D
Roadie
drozdse1Roadie
asked in Community Articles and Got a Question?

Set data type of a new added field (Expression Evaluator)

How can I change the type of a new added field in the Expression Evaluator? Because if I add a field like this: Then the field is defined as a string. But I need to have it as DATETIMEI know I could just not add this field but my goal is to make the pipeline creating not existing tables if needed. So I need to declare all attributes, even they are empty. Thanks

6
D
Roadie
3 days ago
som
Fan
somFan
posted in Community Articles and Got a Question?

Include older sdc logs(sdc.log & sdc-1.log extension) in Support Bundle

Previously on all data collector below 5.X version , The behaviour was that all SDC logs were included in the Support Bundle. Now  only the current log(sdc.log) is  included and sdc-*.log logs are missing.The solution for the fix was applied post 5.4 version onwards.

0
som
Fan
3 days ago
som
Fan
somFan
posted in Community Articles and Got a Question?

View list of jars that has been loaded during SDC start up

We can see the list of jars that has been loaded during SDC start up by below configuration,If using Cloudera Manager, Please add the following to the Data Collector Advanced Configuration Snippet (Safety Valve) for sdc-env.sh configuration and restart SDC: export SDC_JAVA_OPTS="-Dstreamsets.bootstrap.debug=true ${SDC_JAVA_OPTS}" This will tell us what jars are being picked up in the classpath when SDC starts.  The output will appear in stdout.log of the Data Collector process. If  we are using standalone SDC, we can add the above snippet directly to sdc-env.sh ($SDC_HOME/libexec)and restart the sdc.

0
som
Fan
3 days ago
D
Roadie
drozdse1Roadie
asked in Community Articles and Got a Question?

DATE type in Oracle origin database is casted into DATETIME on Snowflake?

Our goal is to sync between Oracle and Snowflake with CDC Oracle. Some of our source tables in Oracle are defined as DATE. So we created this tables also in Snowflake with the same type, DATE.  But now I get this error message:  But in this case the field ACTIVITY_TIME is DATE on Oracle and Snowflake.Later I noticed that we don’t have any problem if I convert all DATE fields into DATETIME attributes on Snowflake. But looking in the documentation, there is nothing mentioned that we have to use DATETIME in such cases. The type DATE is explicitly mentioned as supported type. Does anyone had the same issues? I mean it’s not so unusual to use the DATE type in a database

1
Bikram
Headliner
3 days ago
catchcharan
Fan
catchcharanFan
asked in Community Articles and Got a Question?

HL7 format

Hi Team, Does StreamSets support HL7 data format? I know there is a python way of doing this. Thanks,Meher

3
Bikram
Headliner
8 days ago
M
Fan
MalathiFan
asked in Community Articles and Got a Question?

SDC Snowflake destination is not connecting

Hi,I am trying to load data into snowflake in SDC. But, I am getting below error. Can you please tell me where I am doing wrong because it is working fine in Transfomer for snowflake for the same snowflake account url. In SDC it is not working if I am using the locator on account tab. SNOWFLAKE_11 - Could not create SQL DataSource: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: JDBC driver encountered communication error. Message: Exception encountered for HTTP request: Certificate for <kxxxxxx.eu-north-1.snowflakecomputing.com> doesn't match any of the subject alternative names: [*.prod3.us-west-2.snowflakecomputing.com, *.us-west-2.snowflakecomputing.com, *.global.snowflakecomputing.com, *.snowflakecomputing.com, *.prod3.us-west-2.aws.snowflakecomputing.com]. Thanks,Malathi

0
M
Fan
11 days ago
D
Opening Band
Dhanashri_BhateOpening Band
asked in Community Articles and Got a Question?

GCP: credential:get function - Are there limitations where this function can be used?

Hello, One of my customer is using StresmSets to access secrets saved in GCP. The SDC setup and GCP secrets setup is correct. For example,  I have used secrets like below successfully in JDBC producer stage’s credential tab and pipeline works fine, picks up latest version of the secret as expected. ${credential:get("gcp", "group@org", "dbuser?latest")}${credential:get("gcp", "group@org", "dbpass?latest")}Now, they want to read a secret for which a json file was uploaded as value, and they want to extract credential information from this json.  I tried to use credential:get in an Expression evaluator but it gave an error. (was planning to add more processors after this to extract userid and password from the json returned by credential:get) How can i use secret like this in my StreamSets pipeline? Thanks! 

6
Bikram
Headliner
11 days ago
Mike Calma
StreamSets Employee
Mike CalmaStreamSets Employee
posted in Community Articles and Got a Question?

ADLS Gen 2 Legacy

[ADLS Gen 2 Changes]:In Data Collector Version 5.5.0 a change was made which altered how the ADLS Gen 2 stage functioned. Previously the ADLS Gen 2 Stage utilized the Hadoop FS to access resources. This change also split the stage into the respective resources they access: ADLS Gen 2 and Azure Blob Storage. For additional information surrounding this change please review the patch notes linked below: Ref 1: Patch NotesRef 2: Available Stage Comparison [Legacy ADLS Gen 2]:Additionally, the ADLS Gen 2 Legacy stage which used Hadoop FS is also still available to be utilized.** It should be noted though that when exporting a pipeline made on a version prior to 5.5 into later Data Collectors that the (Legacy) tag will not be present on the stage information. However, the description of the stage will still relay the method with which the resource is accessed. When the (Legacy) stage is selected in a current pipeline it can be identified as seen in the image below  

0
Mike Calma
StreamSets Employee
14 days ago
blakeorsak
Fan
blakeorsakFan
posted in Community Articles and Got a Question?

Excessively large engine-$$.out file in the /tmp directory

The Issue:  Data Collector has created a file in the /tmp directory that is growing to an excessive size and is taking up space.  The purpose of this file is to serve as a location for stdout from a running Data Collector. Under normal circumstances, this file does not receive much data and should not grow to a large size. Specifically, the rotatelogs is used to limit the size of this file to ~10mb.  However, there are a few instances where this file will receive more data than usual, or grow to an excessively large size.  Adding certain Java parameters to your $SDC_HOME/libexec/sdc-env.sh file can result in extra output being sent to this /tmp file. This output is often useful for debugging, but can fill the file up quickly. An example of this might be the -Djavax.net.debug=ssl,handshake parameter used to debug SSL connections.  If the rotatelogs utility is not currently available on your machine, the /tmp/engine-$$.out file may continue to grow past the limited size of ~10mb.  The So

0
blakeorsak
Fan
14 days ago
Francisco
Fan
FranciscoFan
asked in Community Articles and Got a Question?

HTTP Processor for POST request clearing header values

Hello! I’m using record header attributes as variables. For example, during pipeline execution, saving ${time:now()} in a header attribute with an Expression Evaluator, to be used in other stages. This is no problem for HTTP GET requests, as the data gets aggregated into existing records. The problem is with HTTP Processor for POST requests, where the response overwrites all existing data from before, body and headers.Is there anyway around this? I’m trying to use this header attribute value that I saved to be used in anohter HTTP POST request after the one I just mentioned, but the value is lost. Thanks in advance!

1
Bikram
Headliner
15 days ago
paul
StreamSets Employee
paulStreamSets Employee
posted in Community Articles and Got a Question?

SDC 5.7.x - Orchestration pipeline calling a Job Template does not function properly

Environment:StreamSets Data Collector (SDC) 5.7.0 & 5.7.1 An orchestration pipeline which uses Start Jobs stages The orchestration stage starts a job using a Job Template The orchestration stage is configured with the Run in Background setting disabled  Issue:In StreamSets Data Collector (SDC) 5.7.0 and 5.7.1, orchestration pipelines that use a Start Jobs stage to start a job using a Job Template do not function properly when Run in Background is disabled. With this configuration, the pipeline should wait for the job to complete before moving to the next stage, but instead the pipeline starts the job and immediately moves on, which can cause unexpected results for downstream jobs. Resolution:There is a fix planned for this issue in the SDC 5.8.0 release. In the short term, if your organization has an active support contact, you can contact StreamSets Support to obtain a patch to resolve the issue.  

0
paul
StreamSets Employee
21 days ago
paul
StreamSets Employee
paulStreamSets Employee
posted in Community Articles and Got a Question?

SCH 3.x - Sharing Settings - Users/Group not showing up in the search box

IssueWhen updating the sharing settings for pipelines, jobs, etc. typing in a user or group that is known to exist in the organization doesn’t show up in the search box.ResolutionThis occurs when there are more than 50 users and/or groups. Control Hub preloads 50 users and groups for the purposes of autocomplete. If a user or group doesn’t show up, you can type in the full username (user@org) or group name (group@org) and click the add button, and it will work.NoteThis issue is specific to legacy StreamSets Control Hub 3.x. In Platform, the search box has been improved and should autocomplete for all users and groups. 

0
paul
StreamSets Employee
28 days ago
blakeorsak
Fan
blakeorsakFan
posted in Community Articles and Got a Question?

Azure_02 Error when creating an Azure Connection in Control Hub

The Issue:  When attempting to create a Connection for an Azure account, you receive the following error message: ADLS_02 - Failed to connect to Azure Data Lake Store: ‘Operation failed: “An HTTP header that’s mandatory for this request is not specified”,400,PUT, <URL>The Solution:  This error could potentially indicate that the Connection type set in Control Hub and the Azure account type do not match. When using a Data Collector version that is higher than 5.4, you are presented with a number of options when creating a new connection to Azure. These options include: Azure Blob Storage Azure Data Lake Storage Gen2 Azure Synapse When this error is encountered, before troubleshooting further it is important to first check that the connection type that was chosen here matches the Azure Account type you are connecting to. This can be verified by checking the Azure Account FQDN. For example: An FQDN like example.dfs.core.windows.net would be used for an Azure Data Lake Storage Gen2 c

0
blakeorsak
Fan
28 days ago
catchcharan
Fan
catchcharanFan
asked in Community Articles and Got a Question?

Certification - Proctor U code

Hi Team, I have paid for white belt certification and received a receipt number as well with payment confirmation for white belt exam.I never received any email from ProctorU which consists of code that I need to enter to proceed to schedule an exam. It’s been 15 days now.Can anyone help on how to proceed with this issue? Thanks,

1
john.durkin
StreamSets Employee
29 days ago
N
Fan
NateFan
asked in Community Articles and Got a Question?

Platform SDK for Python Version 5.2.1 pipeline.committer return

I am using the python SDK and I when I use the method “.committer” or “.last_modified_by” on a pipeline object, I am expecting to get back an email that I am able to see under “Committed By” on the platform. However, I get back a long string that I can’t make out:sudo code of what I am doing: sch = ControlHub(credential_id="cred_id,token = token)print(sch.pipelines.get(commit_id = commit_id).last_modified_by) and instead of getting a string of an email returned, I am getting something like this returned:a83b3195-61f4-11ee-a6e4-…..@b5f95dcd-5f00-11ec-b405-…… Would you happen to know how I can turn this string into a valid email? I couldn’t find anything on this topic in the docs.  

1
M
Roadie
29 days ago
john.mcavoy
StreamSets Employee
john.mcavoyStreamSets Employee
asked in Community Articles and Got a Question?

Kafka Stage fails with error "Request joining group due to: rebalance failed due to 'null' (DisconnectException)"

IssueA Kafka stage fails to connect to the broker due to the follow error stack trace:INFO ConsumerCoordinator - (Re-)joining groupINFO NetworkClient - Disconnecting from node xxx due to socket connection setup timeout. The timeout value is 8424 ms.INFO ConsumerCoordinator - Group coordinator is unavailable or invalid due to cause: null. isDisconnected: true. Rediscovery will be attempted.2023-09-27 13:07:58,776 INFO ConsumerCoordinator - Request joining group due to: rebalance failed due to 'null' (DisconnectException)BackgroundThis specific Kafka Client exception appears when the Kafka Client’s socket-connection-setup-timeout-max-ms is set too low, causing it to attempt to re-join the group.The maximum amount of time the client will wait for the socket connection to be established. The connection setup timeout will increase exponentially for each consecutive connection failure up to this maximum. SolutionWhen this property is set too low, the Kafka Client will time out during the

0
john.mcavoy
StreamSets Employee
29 days ago
paul
StreamSets Employee
paulStreamSets Employee
posted in Community Articles and Got a Question?

StreamSets Control Hub rpm package for Red Hat Enterprise Linux 8

QuestionThere are rpm packages for Red Hat 6 and 7, but not for Red Hat 8? AnswerStreamSets has tested the RHEL 7 package on 8 and certified it for use. There are no plans to release a separate rpm package specifically labeled for RHEL 8.

0
paul
StreamSets Employee
1 month ago
P
Fan
Priya151997Fan
asked in Community Articles and Got a Question?

After S3 Event need to capture file name in expression evaluator

Hi All,we are new in streamsets !!In our usecase we are uploading file in s3 which predefined name as abcd_sggshs_31-04-2018.csvwe need to capture this above parameter which are in our file name,we are generating events after uploading file in s3,using expression evaluator we are trying to capture this parameter in json file in s3.but we are not getting desired output.below are the expression we are using please suggest the correct expression to get the file name.   

8
R
StreamSets Employee
1 month ago
som
Fan
somFan
asked in Community Articles and Got a Question?

Jobs do not carry over offset value when migrate from one server to another.

The offset is always stored in the Control Hub (SCH). The Data Collector (SDC) stores the offset to the offset.json file and sends the information to the SCH periodically. When 'Enable Failover' is checked, the offset is migratable i.e the SCH can start a job on a different SDC (SDC_2) with the last-saved offset (from the previous SDC - SDC_1, which is not available anymore). If 'Enable Failover' is not checked, the offset is tied to a specific SDC ID and it does not migrate i.e the SCH cannot start a job with the last-saved offset from another SDC (SDC_1) on a different SDC (SDC_2). On a new SDC, the job will run the pipeline from the beginning. Failover needs to be enabled for the offset to be migrated.

0
som
Fan
1 month ago
K
Fan
kushal_upadhyayFan
asked in Community Articles and Got a Question?

Reading avro file from a bucket similar to S3 and trying to convert into parquet

I am reading an avro file from a custom cloud similar to S3 and trying to conver it into parquet file using whole file evaluator but its giving an error  Record1-Error Record1 CONVERT_01 - Failed to validate record is a whole file data format : java.lang.IllegalArgumentException: Record does not contain the mandatory fields /fileRef,/fileInfo,/fileInfo/size for Whole File Format. : (View Stack Trace...   

3
gccoulter
StreamSets Employee
1 month ago
gccoulter
StreamSets Employee
gccoulterStreamSets Employee
posted in Community Articles and Got a Question?

Converting an AVRO file (with Schema) to Parquet, using Whole File Transformer

Here we will be covering how to convert an AVRO file with Schema, to a Parquet file.STEP 1Set your pipeline up as follows;Directory - Whole File Transformer - Local FS STEP 2Configure your Directory Origin as follows; GENERAL TAB FILES TAB Please ensure you make the amendments necessary to certain fields to suit your own set up, e.g. File Name Pattern. Keep the rest as in the screenshot. POST PROCESSING TAB DATA FORMAT TAB STEP 3Configure Whole File Transformer stage as follows; GENERAL TAB JOB TAB*amend file directory as necessary AVRO TO PARQUET TAB If you preview the pipeline, ensure to select the following boxes, which will allow you to click in to each record and schema;  You should now have a .parquet file from your original AVRO.

0
gccoulter
StreamSets Employee
1 month ago

Badge winners

Show all badges
Terms & Conditions