Skip to main content

StreamSets Platform

Get inspired and gain all the knowledge you need

1238 Topics
southwr62
Fan
southwr62Fan
asked in Community Articles and Got a Question?
Start Jobs processor not passing string as a string

I am trying to create a master flow that calls another pipeline but am having issues in that when I pass parameters, they seem to be converted to values, or at least they lose the quote marks {      "TDate": "2023-09-17",      "FDate": "2023-09-14"}In the status window of the job called, they show up as  2023-09-17 and 2023-09-14, instead of “2023-09-17” and “2023-09-14”, and also update the job parameters without the quotes  If I try and calculate the dates {      "TDate": "${record:value('/TDateY')}",      "FDate": "${record:value('/FDateY')}"}I get the following errorSTART_JOB_05 - Failed to parse runtime parameters for job ID: START_JOB_05 - Failed to parse runtime parameters for job ID: 6f86b611-bfbe-4882-a47f-3aa41f0daeab:0f246ae0-4677-11ec-a298-93161f808487, error: com.fasterxml.jackson.core.JsonParseException: Unexpected character (' ' (code 160)): was expecting double-quote to start field name at [Source: (String)"{       "TDate": "2023-09-17",       "FDate": "2023-09-17" }";

411
Bikram
Headliner
S
Fan
Sankar ArunachalamFan
posted in Share your Best Practices
How to do CRUD operations via streamsets ?

How to do CRUD operations via streamsets ?Source- IBM DB2Destination - Azure Postgresql DB.How can be done it? What is best way to accomplish this?

271
Bikram
Headliner
S
Roadie
skyRoadie
asked in Community Articles and Got a Question?
StreamSets Full RPM package download

I would like to download StreamSets Full RPM package of version 3.14.0, but I am unable to do so through the official website. Could someone please send me a copy? I would be very grateful. My email address is 1028259318@qq.com. Thank you in advance.

702
S
Roadie
S
Fan
sgrewalFan
asked in Community Articles and Got a Question?
Excel File as Origin Source using HTTPS

I have a bunch of excel (xls) files that are hosted online (HTTPS) and periodically updated. I want to be able to store the excel file to ADLS Gen2. What is the best way to approach this? Data SourceNo authentication/authorization needed. HTTPS XLS formatEnvironment StackStreamsets Azure Shop

15814
Bikram
Headliner
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in StreamSets Academy
I want to correct the datatypes of all of my input field values ?

I WANT TO WRITE ALL THE FIELD VALUES THAT CONTAINS Blank Values INTO NULL I used these expression for to write specific one field = field ${str:length(record:value('/VENDOR_NAME_ALT')) != 0 ? record:value('/VENDOR_NAME_ALT') : 'null'}but i want to use all  fields  that contains blank values means feed with  NULL . any other processor can replace these scenario

221
Bikram
Headliner
S
Fan
Sankar ArunachalamFan
posted in Share your Best Practices
JSON transformation - How can be done it via the streamsets

Hi team,I need to do  transform the data to JSON format after I fetching the data from PostgreSQL DB. Can you help me on this activity? what processor should I use and how can be done it via streamsets?

261
Bikram
Headliner
Mihir Kale
Roadie
Mihir KaleRoadie
asked in Community Articles and Got a Question?
Reading stage libraries from SDK

Requirement: I want to create an upgraded version of new deployment from an old deployment using SDK. The 2 deployments may have different set of stage libraries depending on their engine versions. So, I want to fetch list of all libraries for each version, compare the old deployment’s libraries with that of the obtained repository comprising upgraded version’s libraries, to check for any deprecated/upgraded versions, and then create the new deployment with the updated stage library list. I want to access all possible stage libraries (common + enterprise) of a particular engine version using SDK. Is there an API to fetch this?

120
Mihir Kale
Roadie
D
Opening Band
Dhanashri_BhateOpening Band
asked in StreamSets Academy
3.x - Job runner API - Getting jobs for a specific pipeline - returns jobs that do not match the criteria

Hello,I’m using Control Hub 3.x ( to replicate the customer’s environment) I need to cleanup old jobs based on the pipeline name (passed as parameter) and n days old. I have created the following pipeline: (also attached)  When I run this API call in browser or via this pipeline, I get many jobs which do not match the criteria.Tried also with creator value and some other fields but facing the same problem. Does this API allow filtering the jobs like this? I can change the pipeline to get all jobs first and then add a stream selector to get the jobs I’m interested in, but I’m trying to limit the API response to only what i need. Can someone help?Dhanashri

310
D
Opening Band
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in Community Articles and Got a Question?
insert blank values in xlxs file into a Null Values ?

i used expression evaluator to give or insert blank record values into Null Values ,how can i achieve this .this is my incoming data from origin. these type field had Blank ,but i want to feed Null ..i used this expression:${str:isNullOrEmpty(record:value('/TYPE_1099'))?NULL:record:value('/TYPE_1099')} but it fails . 

6811
lakshmi_narayanan_t
Discovered Fame
Anmuthu
Fan
AnmuthuFan
asked in Community Articles and Got a Question?
Generating blank output files

I have a Kafka topic as a source and the sink as a delimited files. I want to generate a blank output file if there is no data in the topic. What is the best way to achieve this.  thanks

483
Bikram
Headliner
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in StreamSets Academy
how to Write XLSX file format in localfs.

My original data is in the XLSX spreadsheet file format. However, I would like to save it locally as an XLSX file format, but I am currently unable to do so. Is there an alternative processor or method I can use to achieve this desired file format conversion?

191
Bikram
Headliner
Y
Discovered Fame
yogesh0590
asked in Community Articles and Got a Question?
How to get header (all colums) field as new column ?

Hi Team,I have scenario where I am reading delimited file from S3, in next step I need to compare header of this file. I am unable to get header as new field along with other attributes. @Bikram @Ranjith P @Rishi Any idea on this ? TIA 

593
Bikram
Headliner
Y
Discovered Fame
yogesh0590
asked in Community Articles and Got a Question?
Join operation in SDC

Hi Team,I need to perform a join operation in StreamSets Control Hub using SDC(Data Collectors). As I checked we don’t have any processor or stage in StreamSets SDC.  Is there any workaround we have in SDC. Below are my datasets,Dataset 1 - CSVDataset 2 - Snowflake table Any way to join Dataset 1 and Dataset 2 in SDC. TIA@Rishi @Bikram  @Maria Vila @XavierV  @AkshayJadhav 

403
Bikram
Headliner
Mihir Kale
Roadie
Mihir KaleRoadie
asked in Community Articles and Got a Question?
Incorrect job metrics in orchestration pipeline

I want to evaluate the job metrics field of orchestrator tasks JSON, which is the output of start job origin in orchestration pipeline. The start job origin contains pipeline which moves 5 records from csv file from directory to snowflake table.I need to confirm from the job metrics field that the pipeline output count equals that of input count.But when I preview the orchestration pipeline, both the input and output records count show 0.Is there any additional configuration required that I have missed? I have attached the preview SS along with the job instance pipeline.

402
prasanna_kumar
Roadie
P
Discovered Fame
Priyanka MynepallyDiscovered Fame
asked in Community Articles and Got a Question?
File upload to HTTP

I have a file in resources directory.I need to upload whole file through HTTP.What options to cnfigure. Able to do it in postman

1133
Y
Fan
S
Roadie
skyRoadie
asked in Community Articles and Got a Question?
download the StreamSets software package

I would like to download the StreamSets software package, but when I accesshttps://archives.streamsets.com/index.html, it shows Error 405 Not allowed. How can I resolve this issue? 

730
S
Roadie
N
Fan
NateFan
asked in Community Articles and Got a Question?
Python SDK - How to detect new commits pt2.

Hello, I had a post yesterday: Python SDK - How to detect new commits. | StreamSets CommunityHowever, I cannot find how to reply with another question to the response I got so I will post it here in a new question: Just to clarify what I am asking; I would like to see all new commits across all pipelines. So, say in the last day, 2 existing pipelines have new commits, and one new pipeline is created. I would like to be able to see a log that reports there 3 pipelines have had new commits made, and what their ID /commit_id is. Would the SDK happen to support some sort of logging mechanism like this? I was not able to find it in the docs.Thank you,-Nate

40
N
Fan
N
Fan
nik_shahFan
asked in Community Articles and Got a Question?
Writing value in Salesforce Master Detail Relationship field

I have a scenario where I need to populate the value of a parent object on its child record which is in a master detail relationship. For instance, I have a Custom Object “Customer Number” which is in Master Detail relationship with Leads. Instead of pass Salesforce ID, I want to pass the Lead.Name (as it is available in Streamsets) to establish the relationship of “Customer Number” record to its parent Lead object. What is the syntax that I need to use?

320
N
Fan
N
Fan
NateFan
asked in Community Articles and Got a Question?
Python SDK - How to detect new commits.

Hello,  I have been looking through the SDK docs looking for a functionality that would let me see all new pipeline commits made in a certain time frame. I wasn’t having much luck finding anything and I was wondering if there are any functions that support this. Thank you, -Nate

242
N
Fan
Douglas R
Fan
Douglas RFan
posted in Community Articles and Got a Question?
MySQL: JDBC_06: Failed to Initialize Connection Pool: Access Denied Error

MySQL: JDBC_06: Failed to Initialize Connection Pool: Access Denied Error ProblemA pipeline which is using a JDBC stage to access a MySQL database generates an error int he following form:START_ERROR: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Access denied for user 'username'@'hostname' (using password: YES)The pipeline is using a configured Connection in Control Hub, and when the Test Connection button is used in the Edit Connection window, the connection succeeds.  Only when it is used in an actual pipeline does it fail.CauseThis behavior occurs when two conditions are met:MySQL is configured to require SSL.  This condition can be met in one of three ways: The MySQL user is configured with the REQUIRE SSL option. The “requireSSL” property has been configured in the JDBC Additional Configuration Properties int he JDBC stage, with the value “true” “requireSSL=true” has been added to the JD

3690
Douglas R
Fan
southwr62
Fan
southwr62Fan
asked in Community Articles and Got a Question?
How to use time:dateAddition

Hi, as a relatvie newcomer to Streamsets, I am trying to use ‘time:dateAddition’ function, in data collector. I am using an engine version 5.6.1, but get the same issues with 5.5.0I am basically trying to use time:Now(), to calculate yesterdays date (or at least a different date), but getting nowhere.I even tried using the example in the docs “${time:dateAddition("MONTH", 2, 2007-10-03T10:15:30)}”but get the error‘EXPR_00 - Invalid expression '${time:dateAddition("MONTH", 2, 2007-10-03T10:15:30)}': com.streamsets.pipeline.api.el.ELEvalException: CTRCMN_0101 - Error parsing expression ${time:dateAddition("MONTH", 2, 2007-10-03T10:15:30)}: javax.servlet.jsp.el.ELException: Encountered "T10", expected one of [".", ">", "gt", "<", "lt", "==", "eq", "<=", "le", ">=", "ge", "!=", "ne", ")", ",", "[", "+", "-", "*", "/", "div", "%", "mod", "and", "&&", "or", "||", "?"]’This function seems to be new with 5.5.0Could someone help please.

3694
Bikram
Headliner
S
Fan
santhoshchallaFan
asked in Community Articles and Got a Question?
Find the status of a job using REST API

Hi,I have triggered one of the job suing REST API in StreamSets and I need to the status of the job if it is failed or successful. I am able to get the status of the job but if the job is failed or successful, I am getting status as Inactive in both success and failure case. I need to get exact status like success or failure and error message in case of job failure. Please let me know how I can I achieve this StreamSets. 

3046
prasanna_kumar
Roadie
S
Fan
SUDHANSHOOFan
asked in Share your Best Practices
Need to Read ADLS files name in nested folders

Hi All,We have a senario where we need to read all file name in ADLS Origin Stage. We have checked out in Streamsets documentation but its not working Its mentioned in Doc - we can access the files with **/<File Pattern>But its not picking up the filename.If anyone have the solution you can share with us. 

392
S
Fan
himanshu1234567
Discovered Fame
himanshu1234567Discovered Fame
asked in Community Articles and Got a Question?
Not able to export all pipelines at once using python sdk

Hi team when i am trying to export all pipelines using python sdk , i am facing below error: {"ISSUES":[{"code":"PIPELINE_STORE_41","message":"Cannot export draft pipelines with plain text credentials stripped"}]}So basically we are not able to export draft pipelines with credentials. Does we have any way where we can export all pipelines whether it is published or draft version. 

562
himanshu1234567
Discovered Fame
AdamK
Fan
AdamKFan
asked in Community Articles and Got a Question?
How can I extend default timeout value?

Hi, I’m using StreamsetsSDK in version 3.12.1.In my code which is run on Teamcity agent when I’m publishing changes to pipeline using publish_pipeline() method I’m getting the following error:requests.exceptions.ConnectionError: HTTPSConnectionPool(host='******', port=******): Max retries exceeded with url: /rest/v1/system/info (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f95dca34820>: Failed to establish a new connection: [Errno 110] Connection timed out'))This might be caused by firewall settings because code works fine when run locally.Is there a way to extend timeout limit? On the other hand doing imports using import_jobs() work fine. 

1082
AdamK
Fan

Badge winners

Show all badges
Terms & Conditions