StreamSets Platform

Get inspired and gain all the knowledge you need

1018 Topics

Sperch
Fan
SperchFan
asked in Community Articles and Got a Question?

Troubleshooting Jython Evaluator Code

I am trying to use a Jython evaluator stage to remove the formatting and code encapsulation to only present the string content to be inserted in the destination stage (Snowflake). I have been searching online for ideas and guidance but have found very little, and most not helpful and/or pertinent to my quandary. An example of the string incoming to the Jython stage:<p style="margin: 0in 0in 0pt;"><span style="color: #000000; font-family: verdana, geneva; font-size: 10pt;">This position is embedded within our Data Analytics practice. Leverage your curiosity and problem-solving skills to explore, discover, and predict patterns contained within data sets for a wide range of government clients. This includes the derivation of clear narratives that help our clients understand their data and how those insights address their research questions.</span></p><p style="margin: 0in 0in 0pt;"> </p><p style="margin: 0in 0in 0pt;"><span style="color: #00000

4
Sperch
Fan
2 hours ago
southwr62
Fan
southwr62Fan
asked in Community Articles and Got a Question?

Calculating a Parameter

Hi,I need to use a calculated date in a pipeline, more specifically I need to make an HTTP call, where a date range of the form of a ‘From’ and ‘To’ date, of the form “2023-09-10”.I can calculate this date as a field in a record using  ${time:extractStringFromDate(time:dateAddition("DAY",-1,time:now()),"yyyy-MM-dd")}  But this does not work in the ‘required data’, what would normally be the request body in a post requestIt seems I can use a parameter in this section as follows {"entity":"user","operation":"Login","userName":null,"fromDate":${FDate},"toDate":${TDate},"toDateVal":"2022-07-21T00:00:00.000Z","fromDateVal":"2022-01-01T00:00:00.000Z"}but when I try to create a parameter that calculates the date, using the above formula, I get an EL expression error Can someone suggest where I am going wrong?I have also tried using start jobs, to pass the dates as a parameter, but that also doesn’t seem to work, I can pass a manually entered date to this ‘child job’ but it seems to conv

1
Bikram
Headliner
1 day ago
S
Fan
shobhit_pathakFan
asked in Community Articles and Got a Question?

DB2 CDC Pipeline

I want to create a CDC pipeline for DB2 Database, as there is no origin present for the implementation,Is there any alternative to perform CDC for this Database?

8
Kartik Medavarapu
Roadie
1 day ago
N
Opening Band
N_aliOpening Band
asked in Community Articles and Got a Question?

504 error - Gateway Timeout.

 i have here an sftp origin that i want to pull the files from do some formatting on it and then push it as a bulk post request to HTTP client destination.i keep getting this error 504 gateway timeout with the files with size 200KB or bigger.it is posting the data on the system but returns this error. i tried setting the read timeout configuration and maximum time out values to a much higher values like 15 mins or so in the HTTP client destination but i keep getting the same error though the data is posted in the destination.what should i do to prevent this error?

0
N
Opening Band
1 day ago
saleempothiwala
Headliner
saleempothiwalaHeadliner
asked in Community Articles and Got a Question?

Record Header Attributes for Oracle Bulkload

I am trying to use Oracle Bulkload Origin to read multiple tables from Oracle database. I want to use the tablenames from source to write to destination as files names. In the preview I can see some attributes but when I try to capture them in Expression Evaluator they appear null.  Here is my config for expression evaluator Is it something to do with oracle version? or data collector version?I am using Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - ProductionVersion 21.3.0.0.0SDC version 5.5.0 I can see the jdbc.tables if i use JDBC Multitable Consumer but I want to use Bulkload.

3
saleempothiwala
Headliner
2 days ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in Community Articles and Got a Question?

how to convert data type of decimal into datetime .

this is my datetime values comes from xlsx file and i want to convert into proper datetime VENDOR_CREATION_DATE COLUMN AS A  decimal data type --->want to change datetime I used field type convertor :-to achieve this but its not working ,any one can provide the configuration   

2
Sperch
Fan
2 days ago
southwr62
Fan
southwr62Fan
asked in Community Articles and Got a Question?

Start Jobs processor not passing string as a string

I am trying to create a master flow that calls another pipeline but am having issues in that when I pass parameters, they seem to be converted to values, or at least they lose the quote marks {      "TDate": "2023-09-17",      "FDate": "2023-09-14"}In the status window of the job called, they show up as  2023-09-17 and 2023-09-14, instead of “2023-09-17” and “2023-09-14”, and also update the job parameters without the quotes  If I try and calculate the dates {      "TDate": "${record:value('/TDateY')}",      "FDate": "${record:value('/FDateY')}"}I get the following errorSTART_JOB_05 - Failed to parse runtime parameters for job ID: START_JOB_05 - Failed to parse runtime parameters for job ID: 6f86b611-bfbe-4882-a47f-3aa41f0daeab:0f246ae0-4677-11ec-a298-93161f808487, error: com.fasterxml.jackson.core.JsonParseException: Unexpected character (' ' (code 160)): was expecting double-quote to start field name at [Source: (String)"{       "TDate": "2023-09-17",       "FDate": "2023-09-17" }";

1
Bikram
Headliner
3 days ago
S
Fan
Sankar ArunachalamFan
posted in Share your Best Practices

How to do CRUD operations via streamsets ?

How to do CRUD operations via streamsets ?Source- IBM DB2Destination - Azure Postgresql DB.How can be done it? What is best way to accomplish this?

1
Bikram
Headliner
3 days ago
S
Fan
skyFan
asked in Community Articles and Got a Question?

StreamSets Full RPM package download

I would like to download StreamSets Full RPM package of version 3.14.0, but I am unable to do so through the official website. Could someone please send me a copy? I would be very grateful. My email address is 1028259318@qq.com. Thank you in advance.

2
S
Fan
3 days ago
S
Fan
sgrewalFan
asked in Community Articles and Got a Question?

Excel File as Origin Source using HTTPS

I have a bunch of excel (xls) files that are hosted online (HTTPS) and periodically updated. I want to be able to store the excel file to ADLS Gen2. What is the best way to approach this? Data SourceNo authentication/authorization needed. HTTPS XLS formatEnvironment StackStreamsets Azure Shop

14
Bikram
Headliner
3 days ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in StreamSets Academy

I want to correct the datatypes of all of my input field values ?

I WANT TO WRITE ALL THE FIELD VALUES THAT CONTAINS Blank Values INTO NULL I used these expression for to write specific one field = field ${str:length(record:value('/VENDOR_NAME_ALT')) != 0 ? record:value('/VENDOR_NAME_ALT') : 'null'}but i want to use all  fields  that contains blank values means feed with  NULL . any other processor can replace these scenario

1
Bikram
Headliner
6 days ago
S
Fan
Sankar ArunachalamFan
posted in Share your Best Practices

JSON transformation - How can be done it via the streamsets

Hi team,I need to do  transform the data to JSON format after I fetching the data from PostgreSQL DB. Can you help me on this activity? what processor should I use and how can be done it via streamsets?

1
Bikram
Headliner
6 days ago
M
Fan
Mihir KaleFan
asked in Community Articles and Got a Question?

Reading stage libraries from SDK

Requirement: I want to create an upgraded version of new deployment from an old deployment using SDK. The 2 deployments may have different set of stage libraries depending on their engine versions. So, I want to fetch list of all libraries for each version, compare the old deployment’s libraries with that of the obtained repository comprising upgraded version’s libraries, to check for any deprecated/upgraded versions, and then create the new deployment with the updated stage library list. I want to access all possible stage libraries (common + enterprise) of a particular engine version using SDK. Is there an API to fetch this?

0
M
Fan
6 days ago
D
Opening Band
Dhanashri_BhateOpening Band
asked in StreamSets Academy

3.x - Job runner API - Getting jobs for a specific pipeline - returns jobs that do not match the criteria

Hello,I’m using Control Hub 3.x ( to replicate the customer’s environment) I need to cleanup old jobs based on the pipeline name (passed as parameter) and n days old. I have created the following pipeline: (also attached)  When I run this API call in browser or via this pipeline, I get many jobs which do not match the criteria.Tried also with creator value and some other fields but facing the same problem. Does this API allow filtering the jobs like this? I can change the pipeline to get all jobs first and then add a stream selector to get the jobs I’m interested in, but I’m trying to limit the API response to only what i need. Can someone help?Dhanashri

0
D
Opening Band
7 days ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in StreamSets Academy

how to covert file and write into localfs as a XLSX file Format.

can you please suggest any other configuration or proseccor to achieve output.i reading my file in directory origin origin is :configuration my destination is localfs:configuration  after  execution of my pipeline :-out put file like - But my requirement is like  how to acive my fie format ,by using any processor jython evaluvator or whole file transformer.

0
lakshmi_narayanan_t
Discovered Fame
7 days ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in Community Articles and Got a Question?

insert blank values in xlxs file into a Null Values ?

i used expression evaluator to give or insert blank record values into Null Values ,how can i achieve this .this is my incoming data from origin. these type field had Blank ,but i want to feed Null ..i used this expression:${str:isNullOrEmpty(record:value('/TYPE_1099'))?NULL:record:value('/TYPE_1099')} but it fails . 

11
lakshmi_narayanan_t
Discovered Fame
7 days ago
Anmuthu
Fan
AnmuthuFan
asked in Community Articles and Got a Question?

Generating blank output files

I have a Kafka topic as a source and the sink as a delimited files. I want to generate a blank output file if there is no data in the topic. What is the best way to achieve this.  thanks

3
Bikram
Headliner
7 days ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in StreamSets Academy

how to Write XLSX file format in localfs.

My original data is in the XLSX spreadsheet file format. However, I would like to save it locally as an XLSX file format, but I am currently unable to do so. Is there an alternative processor or method I can use to achieve this desired file format conversion?

1
Bikram
Headliner
7 days ago
Y
Discovered Fame
yogesh0590
asked in Community Articles and Got a Question?

How to get header (all colums) field as new column ?

Hi Team,I have scenario where I am reading delimited file from S3, in next step I need to compare header of this file. I am unable to get header as new field along with other attributes. @Bikram @Ranjith P @Rishi Any idea on this ? TIA 

3
Bikram
Headliner
7 days ago
Y
Discovered Fame
yogesh0590
asked in Community Articles and Got a Question?

Join operation in SDC

Hi Team,I need to perform a join operation in StreamSets Control Hub using SDC(Data Collectors). As I checked we don’t have any processor or stage in StreamSets SDC.  Is there any workaround we have in SDC. Below are my datasets,Dataset 1 - CSVDataset 2 - Snowflake table Any way to join Dataset 1 and Dataset 2 in SDC. TIA@Rishi @Bikram  @Maria Vila @XavierV  @AkshayJadhav 

3
Bikram
Headliner
7 days ago
M
Fan
Mihir KaleFan
asked in Community Articles and Got a Question?

Incorrect job metrics in orchestration pipeline

I want to evaluate the job metrics field of orchestrator tasks JSON, which is the output of start job origin in orchestration pipeline. The start job origin contains pipeline which moves 5 records from csv file from directory to snowflake table.I need to confirm from the job metrics field that the pipeline output count equals that of input count.But when I preview the orchestration pipeline, both the input and output records count show 0.Is there any additional configuration required that I have missed? I have attached the preview SS along with the job instance pipeline.

2
prasanna_kumar
Roadie
8 days ago
P
Discovered Fame
Priyanka MynepallyDiscovered Fame
asked in Community Articles and Got a Question?

File upload to HTTP

I have a file in resources directory.I need to upload whole file through HTTP.What options to cnfigure. Able to do it in postman

3
Y
Fan
8 days ago
S
Fan
skyFan
asked in Community Articles and Got a Question?

download the StreamSets software package

I would like to download the StreamSets software package, but when I accesshttps://archives.streamsets.com/index.html, it shows Error 405 Not allowed. How can I resolve this issue? 

0
S
Fan
8 days ago
N
Fan
NateFan
asked in Community Articles and Got a Question?

Python SDK - How to detect new commits pt2.

Hello, I had a post yesterday: Python SDK - How to detect new commits. | StreamSets CommunityHowever, I cannot find how to reply with another question to the response I got so I will post it here in a new question: Just to clarify what I am asking; I would like to see all new commits across all pipelines. So, say in the last day, 2 existing pipelines have new commits, and one new pipeline is created. I would like to be able to see a log that reports there 3 pipelines have had new commits made, and what their ID /commit_id is. Would the SDK happen to support some sort of logging mechanism like this? I was not able to find it in the docs.Thank you,-Nate

0
N
Fan
9 days ago
N
Fan
nik_shahFan
asked in Community Articles and Got a Question?

Writing value in Salesforce Master Detail Relationship field

I have a scenario where I need to populate the value of a parent object on its child record which is in a master detail relationship. For instance, I have a Custom Object “Customer Number” which is in Master Detail relationship with Leads. Instead of pass Salesforce ID, I want to pass the Lead.Name (as it is available in Streamsets) to establish the relationship of “Customer Number” record to its parent Lead object. What is the syntax that I need to use?

0
N
Fan
9 days ago

Badge winners

Show all badges
Terms & Conditions