Got a Question?

Can't find what you're looking for? Ask it here!

  • 563 Topics
  • 1,368 Replies

563 Topics

R
Fan
RohithFan
asked in Got a Question?

Gpg encryption not working

I am doing gpg encryption in streamsets pipeline using shell stage with the below command.gpg -r <keyid> -e <filename>Its working fine in the lower environments without any issues but when I try the same in the higher environments it is throwing error as gpg: skipped : no public key even though I imported the public key in server. can someone help me with this 

0
R
Fan
19 days ago
Y
Roadie
yogesh0590Roadie
asked in Got a Question?

Importing Python module in Jython Stage

I am trying to use Jython Stage to process the JSON data as it is bit complex JSON.I used below code to import python module but getting error, Code : try: sdc.importLock() import sys sys.path.append('/python/sdc-packages') import json_normalizefinally: sdc.importUnlock() Error :com.streamsets.pipeline.api.StageException: SCRIPTING_06 - Script error while processing batch: javax.script.ScriptException: ImportError: No module named json_normalize in <script> at line If you can advise.

2
Bikram
Headliner
22 days ago
P
Roadie
Priyanka MynepallyRoadie
asked in Got a Question?

Kafka doubt

Can someone please help me with solution for this? I have a kafka processor in origin and jython evaluator next to it.I need to check incoming source rows from kafka using jython to stop pipeline if no more kafka-messages. JYTHON:Init Script: state['first_batch'] = "true"if (state['first_batch'] == "false" and len(records) == 0):  sdc.log.info("No more Kafka messages to consume. Stopping pipeline. See ya!")  sdc.toEvent(sdc.createEvent("no-more-messages", 0))for record in sdc.records:  try:    sdc.output.write(record)  except Exception as e:    # Send record to error    sdc.error.write(record, str(e))if (state['first_batch'] == "true" and len(records) > 0):  state['first_batch'] = "false Sometimes Jython works, sometimes it keeps running in infinite time even if no events.If I trigger job once, it works fine, 2nd time also fine, But from third time in a day I see this problem that pipeline doesnt stop even if kafka has no events.

0
P
Roadie
22 days ago
S
Fan
songforwhoeverFan
asked in Got a Question?

Edit Pipeline Stage Config & Republish

Facing some issues when trying to update an existing CH pipeline and re-publish as a new version.I am trying to update a particular stage within the pipeline and then re-publish as a new version. When I publish the pipeline, the version number is correct, it’s iterated by 1 but the pipeline is empty. How do I carry through all the contents of the previous version? (stages etc..)

1
saleempothiwala
Opening Band
23 days ago
S
Fan
songforwhoeverFan
asked in Got a Question?

List Pipeline Stages

Is it possible to list the stages of an existing pipeline once you have the existing pipeline in the builder?

6
S
Fan
23 days ago
Y
Roadie
yogesh0590Roadie
asked in Got a Question?

JSON Flattening with Nested elements

Hello Team,Could you please assist me with below JSON flattening, [ { "ID1": "Test_value", "ID2": "Test_value2", "ID3": { "ID3_1": 1, "ID3_2": 2, "MAIN_1": { "OUTPUT": [ { "OUT1": "1", "OUT2": "2", "OUT3": "3", "OUT4": "4" }, { "OUT1": "6", "OUT2": "7", "OUT3": "8", "OUT4": "9" }, { "OUT1": "10", "OUT2": "11", "OUT3": "12", "OUT4": "13" } ] }, "MAIN_2": { "OUTPUT": [ { "OUT1": "1", "OUT2": "2", "OUT3": "3", "OUT4": "4" }, { "OUT1": "6", "OUT2": "7", "OUT3": "8", "OUT4": "9" }, { "OUT1": "10", "OUT2": "11", "OUT3": "12", "OUT4": "13" } ] }, "MAIN_3":

12
Fredlau
Fan
23 days ago
A
Fan
ashish_sanghviFan
asked in Got a Question?

pipeline with multitable consumer does not work with altering table from mysql command line

Why, while jdbc pipeline with multitable consumer is running, I can not alter table from command line mysql?

0
A
Fan
24 days ago
jarquero
Fan
jarqueroFan
asked in Got a Question?

Post-processing for whole files format

Hi everybody! I want to remove files  (whole file data format) from a origin stage (GCS stage) after processing it, but when I activate thge post-processing option and set it to "Delete" and try to start the associated job to the pipeline a have the Error:GCS_11 - Error validating postprocessing options. If 'Whole File' data format is used, neither 'Delete' nor 'Archive -> Move into ...' can be used.There are any way to do that directly from a pipeline or the best practice is to add a shell script at the end of the pipeline to remove the files from command line with google CLI commands after processing them? 

1
Edy.Manzin
StreamSets Employee
24 days ago
R
Roadie
rayyan143786Roadie
asked in Got a Question?

is there a way to make the sql server cdc client to pull changes from source at less frequency?

We have huge amount of load at sql server db side due to streamsets and we want to reduce the same. We are using Streamsets sql server cdc client to pull data from 500 sql server tables to Snowflake. Can you please help us below questions:Is there a way to reduce frequency with which stream sets cdc client connect to sql server? We have used queries per second parameter as 1 to reduce the number of queries running at source. please let me know if we can do anything? And we see huge number of select max($__$seqval) running. Is this expected? Thanks!

1
Dimas Cabré i Chacón
StreamSets Employee
26 days ago
kumarankit2594
Fan
kumarankit2594Fan
asked in Got a Question?

Disable Stage(s) in Data Collecor Pipeline

Hello Community,Is there a way to disable specific stage or group of stages of pipeline while previewing the data flow? Suppose I want to check the from origin to first 3 processors but not the rest 4 processors in the flow. How would I do that? Thanks in advance!

1
Dimas Cabré i Chacón
StreamSets Employee
26 days ago
V
Fan
vibha_mardaFan
asked in Got a Question?

Transformer processing power

. 

1
Bikram
Headliner
28 days ago
Y
Roadie
yogesh0590Roadie
asked in Got a Question?

"Disable Read Ahead Stream" config issue

Hello Team,We have pipeline where we have ,SFTP  → S3It is working only when “Disable Read Ahead Stream”  is enable but it is failing while it is disabled.I am facing below error,Unable to write object to Amazon S3, reason : com.amazonaws.SdkClientException: Unable to execute HTTP request: No such file Could you please assist with this error ? 

2
Bikram
Headliner
28 days ago
P
Fan
Priya151997Fan
asked in Got a Question?

After S3 Event need to capture file name in expression evaluator

Hi All,we are new in streamsets !!In our usecase we are uploading file in s3 which predefined name as abcd_sggshs_31-04-2018.csvwe need to capture this above parameter which are in our file name,we are generating events after uploading file in s3,using expression evaluator we are trying to capture this parameter in json file in s3.but we are not getting desired output.below are the expression we are using please suggest the correct expression to get the file name.   

2
Bikram
Headliner
30 days ago
Dash
Headliner
DashSenior Technical Evangelist and Developer Advocate at Snowflake
asked in Got a Question?

What is the StreamSets Summer ‘21 free service tier?

Is the StreamSets Summer ‘21 free service tier completely equivalent to the Data Collector 3.x download?

2
saisumanth
Fan
1 month ago
D
Fan
dhanraj_shindeFan
asked in Got a Question?

API Error The engine response data size [69 MB] exceeds the configured limit [50 MB]

Error:The engine response data size [69 MB] exceeds the configured limit [50 MB] for the API. Try decreasing the origin batch size, or configure the web browser to use direct engine REST APIs to communicate with engines. It works fine when we run the job,but pop’s the above error when try to preview the pipeline. The api I am using have a single batch with large data. 

1
AkshayJadhav
StreamSets Employee
1 month ago
A
Fan
aditya_ghongadeFan
asked in Got a Question?

Unable to send email when alert is triggered

I tried to send email from notifications tab after alert is triggered. Also made the following changes in the config file as well as email-password.txt file. Still the mail is not being sent. I don’t know where I am going wrong. Please suggest me the changes needed.  (Here opentrash is fake email we are using to check if we are receiving mail or not.)config_dc 

3
AkshayJadhav
StreamSets Employee
1 month ago
D
Fan
dhanraj_shindeFan
asked in Got a Question?

Reader exceeded the read limit '1048576' For API call.

I am using http client for API call.Error msg :HTTP_00 - Cannot parse record. HTTP-Status: 200 Reason: com.streamsets.pipeline.api.ext.io.OverrunException: Reader exceeded the read limit '1048576'  API is working if tried on browser.  

2
Bikram
Headliner
1 month ago
M
Fan
MaheshFan
asked in Got a Question?

Bulk ingestion

I want to ingest 10 tables from Oracle can i ingest them with one pipeline or do i have to create 10 pipelines.if we can’t ingest them at once can do we have to create 10 jobs? or is there any alternate option (like master job or for each loop in azure)   

6
Bikram
Headliner
1 month ago
Lav Pandey
Fan
Lav PandeyFan
asked in Got a Question?

Don't Want to Parse JOSN Nested Object while Loading to Snowflake Stage

Hi, I am new to using Streamsets, I am facing one challenge, Current DesignBelow is the JSON Sample Input{    "NAME": "TOM",    "ID": "101",    "DATE": "29-12-2022",    "ADDRESS": { "CITY": "NYC",  "REGION": "NYC",   "COUNTRY": "USA", "PIN": "1234" }} I want Output to be as :NAME, ID, DATE, ADDRESSTOM,101,29-12-2022,{ "CITY": "NYC","REGION": "NYC","COUNTRY": "USA","PIN": "1234"} My Current design is giving me output as :NAME, ID, DATE, CITY, REGION, COUNTRY, PINTOM,101,29-12-2022,NY,NYC,USA,1234 Note :  Current design is a handover code. Here, I don't want to parse the nested JSON, just want to represent it as it is.

8
Lav Pandey
Fan
1 month ago
Ritantara
Fan
RitantaraFan
asked in Got a Question?

Issue in configuration of SFTP destination

USE CASE- Read files from HDFS,SFTP the files and delete the files from HDFS on successful SFTP .I am not able to configure the sFTP destination  .It throws validation error .With the same configuration details ,I was able to configure SFTP as source...The issue persists only around SFTP destination.I am using Resource URL,Username and Private key to configure it .I am certain that there is no issue with the credentials because it is working with SFTP source ..  Can you please highlight if SFTP Destination demands something else..

1
Bikram
Headliner
1 month ago
D
Fan
dinesh
asked in Got a Question?

Failed to read response for messaging/rest/v1/events. connection time out issue

i have been getting time out issue between start job processor and child job triggered from start job. i removed HTTPs time out also but still not sure where and why the error is coming up.[thread:eventHandlerExecutor-pool-4-thread-34] [stage:] ERROR EventClientImpl - Failed to read response for messaging/rest/v1/events. Events: javax.ws.rs.ProcessingException: javax.net.ssl.SSLException: Read timed out    at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:287)    at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:252)    at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:684)    at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:681)    at org.glassfish.jersey.internal.Errors.process(Errors.java:315)    at org.glassfish.jersey.internal.Errors.process(Errors.java:297)    at org.glassfish.jersey.internal.Errors.process(Errors.java:228)    at org.glassfish.jersey.process.internal.Req

1
Brenna
StreamSets Employee
1 month ago
R
Fan
rcanvasFan
asked in Got a Question?

Need help to determine what causes these errors

 Hello,I am looking for insight to what root cause could generate this error in my pipeline. As of a week ago this pipeline which is scheduled to run every 10 mins was running with no issues. On the 16th of December 2022 this pipeline began producing the below error messages. The pipeline grabs files from a local FS and sends them to S3. Under the state history we are given the following for one of the runs: com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0791 - Pipeline lifecycle event stage run error: com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0792 - Pipeline lifecycle event stage generated error record: SHELL_002 - Script execution timed out Any insight on what might be causing this issue would be greatly appreciated. Thanks, (full log file for pipeline error)[runner:] [stage:] ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINE

1
albertfc
Roadie
1 month ago
H
Fan
hari_koramutlaFan
asked in Got a Question?

CDC: How to configure multiple tables for CDC in a single pipeline in a single destination node/table?

How to configure multiple tables (rather configuring each field/attribute) for CDC in a single pipeline to a single destination node/table?Ex: I have 5 tables which are different in terms of no of columns, column datatypes, field data size etc.If I need to build CDC for all the 5 tables which gives table specific data without mapping/configuring field by field in a single pipeline in a single destination node (that is table). 

1
Bikram
Headliner
1 month ago
J
Fan
jyuanFan
asked in Got a Question?

In SDK for python, how to review the properties of all versions of a pipeline, and associated jobs

I am just starting to use the SDK for python (version 3.12.1). I retrieve all pipelines. But, for each pipeline, I can get only the latest version (even though it may be a draft version).I’d like to get all versions of a pipeline. This way, I can see which version is being used by a job (or multiple jobs). If I can only see the latest version, it is very likely that I can’t find any related job.Can I retrieve all versions of a pipeline?Thanks a lot for your attention!

3
wilsonshamim
StreamSets Employee
1 month ago
Y
Roadie
yogesh0590Roadie
asked in Got a Question?

Switch to another data collector in case of any error

Hello Team,Could you please assist for a solution where we can switch to Data Collector A to Data Collector B for a Job if Job having error in between and go for retry.

7
R
StreamSets Employee
1 month ago

Badge winners

Show all badges
Terms & Conditions