Community Articles and Got a Question?

Can't find what you're looking for? Ask it here or check out the Community articles

  • 906 Topics
  • 2,170 Replies

906 Topics

S
Fan
sudarshan_shinde1Fan
asked in Community Articles and Got a Question?

Get list of pipeline name using Python SDK

Hi Team,I am using below mentioned method to get list of all pipelines.But it is returning all the values like version, commit id.I want only pipeline name.sch = ControlHub(credential_id, token)pipeline_list = sch.pipelines.get_all() I tried sch.pipeline.name but it is not supporting.

4
Kirti
StreamSets Employee
6 hours ago
J
Fan
Jhon01Fan
asked in Community Articles and Got a Question?

The API functions correctly in Speakatoo POSTMAN but is not working within my integrated application.

"I've noticed that the Speakatoo API functions properly when tested in POSTMAN, but it's not performing as expected within my integrated application. I'm seeking assistance to understand and address the issue with the API integration in my application."

1
Bikram
Headliner
4 days ago
southwr62
Fan
southwr62Fan
asked in Community Articles and Got a Question?

Problem with Pagination

Hi, I am using the HTTP client origin, and since I need to load a lot of records, I need to use pagination.my url ends with ‘auditLogs?offset=${startAt}&pagesize=5&isascorder=false&order=id’ cannot give more for confidentiallity reasons. This is a post requestwhen I set pagination to offset or page number, all I get is the first 5 records, offset is actually used to denote the start record of the data returned, not a page in this instance.If I set pagesize to 1, which I think is how I need to do this, I only get 1 result it doesn’t seem to continue, I have tried with both page and offset set as the pagination method.any ideas on what is wrong are welcomed

1
M
Fan
6 days ago
abibatoul20
Fan
abibatoul20Fan
asked in Community Articles and Got a Question?

Enable StreamSets Control Hub

Bonsoir, j’ai installé StreamSets Data Collector mais je veux activer le control hub. Quand j’essaie ça me génère l’erreur suivante : DPM Login failed, status code '403': {"message":"Authentication failed"} ( View Details... )

1
Kirti
StreamSets Employee
7 days ago
swapniel22
Fan
swapniel22Fan
asked in Community Articles and Got a Question?

Seeking Assistance with Column Renaming and SDC Field Update in StreamSets Pipeline. SDK

Hello,I am currently running a pipeline that reads data from Oracle Bulkload as the source and utilizes JDBC Producer for PostgreSQL as the destination.Now, I am facing an issue where I need to rename a column from 'FIRST_NAME' to 'second_name' in the destination table, which is in PostgreSQL. I've attempted the following methods:1. Using Oracle Bulkload with a Field Renamer stage followed by JDBC Producer.   - Error: JDBC_07 - Invalid column mapping from field '/Second_name' to column '/first_name'. 2. Using JDBC Consumer Table with a Field Renamer stage followed by JDBC Producer.   - Error: JDBC_07 - Invalid column mapping from field '/Second_name' to column '/first_name'. 3. Using JDBC Consumer Table with a Field Renamer stage followed by JDBC Producer.   - Error: com.streamsets.pipeline.api.base.OnRecordErrorException: JDBC_90 - Record doesn't have any columns for table 'table1'.   - Note: For option 3, I left the Field to Column Mapping empty in JDBC Producer. ……………………………………………………

6
swapniel22
Fan
7 days ago
Sperch
Fan
SperchFan
asked in Community Articles and Got a Question?

Troubleshooting Jython Evaluator Code

I am trying to use a Jython evaluator stage to remove the formatting and code encapsulation to only present the string content to be inserted in the destination stage (Snowflake). I have been searching online for ideas and guidance but have found very little, and most not helpful and/or pertinent to my quandary. An example of the string incoming to the Jython stage:<p style="margin: 0in 0in 0pt;"><span style="color: #000000; font-family: verdana, geneva; font-size: 10pt;">This position is embedded within our Data Analytics practice. Leverage your curiosity and problem-solving skills to explore, discover, and predict patterns contained within data sets for a wide range of government clients. This includes the derivation of clear narratives that help our clients understand their data and how those insights address their research questions.</span></p><p style="margin: 0in 0in 0pt;"> </p><p style="margin: 0in 0in 0pt;"><span style="color: #00000

4
Sperch
Fan
12 days ago
southwr62
Fan
southwr62Fan
asked in Community Articles and Got a Question?

Calculating a Parameter

Hi,I need to use a calculated date in a pipeline, more specifically I need to make an HTTP call, where a date range of the form of a ‘From’ and ‘To’ date, of the form “2023-09-10”.I can calculate this date as a field in a record using  ${time:extractStringFromDate(time:dateAddition("DAY",-1,time:now()),"yyyy-MM-dd")}  But this does not work in the ‘required data’, what would normally be the request body in a post requestIt seems I can use a parameter in this section as follows {"entity":"user","operation":"Login","userName":null,"fromDate":${FDate},"toDate":${TDate},"toDateVal":"2022-07-21T00:00:00.000Z","fromDateVal":"2022-01-01T00:00:00.000Z"}but when I try to create a parameter that calculates the date, using the above formula, I get an EL expression error Can someone suggest where I am going wrong?I have also tried using start jobs, to pass the dates as a parameter, but that also doesn’t seem to work, I can pass a manually entered date to this ‘child job’ but it seems to conv

1
Bikram
Headliner
13 days ago
S
Fan
shobhit_pathakFan
asked in Community Articles and Got a Question?

DB2 CDC Pipeline

I want to create a CDC pipeline for DB2 Database, as there is no origin present for the implementation,Is there any alternative to perform CDC for this Database?

8
Kartik Medavarapu
Roadie
13 days ago
N
Opening Band
N_aliOpening Band
asked in Community Articles and Got a Question?

504 error - Gateway Timeout.

 i have here an sftp origin that i want to pull the files from do some formatting on it and then push it as a bulk post request to HTTP client destination.i keep getting this error 504 gateway timeout with the files with size 200KB or bigger.it is posting the data on the system but returns this error. i tried setting the read timeout configuration and maximum time out values to a much higher values like 15 mins or so in the HTTP client destination but i keep getting the same error though the data is posted in the destination.what should i do to prevent this error?

0
N
Opening Band
13 days ago
saleempothiwala
Headliner
saleempothiwalaHeadliner
asked in Community Articles and Got a Question?

Record Header Attributes for Oracle Bulkload

I am trying to use Oracle Bulkload Origin to read multiple tables from Oracle database. I want to use the tablenames from source to write to destination as files names. In the preview I can see some attributes but when I try to capture them in Expression Evaluator they appear null.  Here is my config for expression evaluator Is it something to do with oracle version? or data collector version?I am using Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - ProductionVersion 21.3.0.0.0SDC version 5.5.0 I can see the jdbc.tables if i use JDBC Multitable Consumer but I want to use Bulkload.

3
saleempothiwala
Headliner
14 days ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in Community Articles and Got a Question?

how to convert data type of decimal into datetime .

this is my datetime values comes from xlsx file and i want to convert into proper datetime VENDOR_CREATION_DATE COLUMN AS A  decimal data type --->want to change datetime I used field type convertor :-to achieve this but its not working ,any one can provide the configuration   

2
Sperch
Fan
14 days ago
southwr62
Fan
southwr62Fan
asked in Community Articles and Got a Question?

Start Jobs processor not passing string as a string

I am trying to create a master flow that calls another pipeline but am having issues in that when I pass parameters, they seem to be converted to values, or at least they lose the quote marks {      "TDate": "2023-09-17",      "FDate": "2023-09-14"}In the status window of the job called, they show up as  2023-09-17 and 2023-09-14, instead of “2023-09-17” and “2023-09-14”, and also update the job parameters without the quotes  If I try and calculate the dates {      "TDate": "${record:value('/TDateY')}",      "FDate": "${record:value('/FDateY')}"}I get the following errorSTART_JOB_05 - Failed to parse runtime parameters for job ID: START_JOB_05 - Failed to parse runtime parameters for job ID: 6f86b611-bfbe-4882-a47f-3aa41f0daeab:0f246ae0-4677-11ec-a298-93161f808487, error: com.fasterxml.jackson.core.JsonParseException: Unexpected character (' ' (code 160)): was expecting double-quote to start field name at [Source: (String)"{       "TDate": "2023-09-17",       "FDate": "2023-09-17" }";

1
Bikram
Headliner
15 days ago
S
Fan
skyFan
asked in Community Articles and Got a Question?

StreamSets Full RPM package download

I would like to download StreamSets Full RPM package of version 3.14.0, but I am unable to do so through the official website. Could someone please send me a copy? I would be very grateful. My email address is 1028259318@qq.com. Thank you in advance.

2
S
Fan
15 days ago
S
Fan
sgrewalFan
asked in Community Articles and Got a Question?

Excel File as Origin Source using HTTPS

I have a bunch of excel (xls) files that are hosted online (HTTPS) and periodically updated. I want to be able to store the excel file to ADLS Gen2. What is the best way to approach this? Data SourceNo authentication/authorization needed. HTTPS XLS formatEnvironment StackStreamsets Azure Shop

14
Bikram
Headliner
15 days ago
M
Fan
Mihir KaleFan
asked in Community Articles and Got a Question?

Reading stage libraries from SDK

Requirement: I want to create an upgraded version of new deployment from an old deployment using SDK. The 2 deployments may have different set of stage libraries depending on their engine versions. So, I want to fetch list of all libraries for each version, compare the old deployment’s libraries with that of the obtained repository comprising upgraded version’s libraries, to check for any deprecated/upgraded versions, and then create the new deployment with the updated stage library list. I want to access all possible stage libraries (common + enterprise) of a particular engine version using SDK. Is there an API to fetch this?

0
M
Fan
18 days ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in Community Articles and Got a Question?

insert blank values in xlxs file into a Null Values ?

i used expression evaluator to give or insert blank record values into Null Values ,how can i achieve this .this is my incoming data from origin. these type field had Blank ,but i want to feed Null ..i used this expression:${str:isNullOrEmpty(record:value('/TYPE_1099'))?NULL:record:value('/TYPE_1099')} but it fails . 

11
lakshmi_narayanan_t
Discovered Fame
19 days ago
Anmuthu
Fan
AnmuthuFan
asked in Community Articles and Got a Question?

Generating blank output files

I have a Kafka topic as a source and the sink as a delimited files. I want to generate a blank output file if there is no data in the topic. What is the best way to achieve this.  thanks

3
Bikram
Headliner
20 days ago
Y
Discovered Fame
yogesh0590
asked in Community Articles and Got a Question?

How to get header (all colums) field as new column ?

Hi Team,I have scenario where I am reading delimited file from S3, in next step I need to compare header of this file. I am unable to get header as new field along with other attributes. @Bikram @Ranjith P @Rishi Any idea on this ? TIA 

3
Bikram
Headliner
20 days ago
Y
Discovered Fame
yogesh0590
asked in Community Articles and Got a Question?

Join operation in SDC

Hi Team,I need to perform a join operation in StreamSets Control Hub using SDC(Data Collectors). As I checked we don’t have any processor or stage in StreamSets SDC.  Is there any workaround we have in SDC. Below are my datasets,Dataset 1 - CSVDataset 2 - Snowflake table Any way to join Dataset 1 and Dataset 2 in SDC. TIA@Rishi @Bikram  @Maria Vila @XavierV  @AkshayJadhav 

3
Bikram
Headliner
20 days ago
M
Fan
Mihir KaleFan
asked in Community Articles and Got a Question?

Incorrect job metrics in orchestration pipeline

I want to evaluate the job metrics field of orchestrator tasks JSON, which is the output of start job origin in orchestration pipeline. The start job origin contains pipeline which moves 5 records from csv file from directory to snowflake table.I need to confirm from the job metrics field that the pipeline output count equals that of input count.But when I preview the orchestration pipeline, both the input and output records count show 0.Is there any additional configuration required that I have missed? I have attached the preview SS along with the job instance pipeline.

2
prasanna_kumar
Roadie
20 days ago
P
Discovered Fame
Priyanka MynepallyDiscovered Fame
asked in Community Articles and Got a Question?

File upload to HTTP

I have a file in resources directory.I need to upload whole file through HTTP.What options to cnfigure. Able to do it in postman

3
Y
Fan
20 days ago
S
Fan
skyFan
asked in Community Articles and Got a Question?

download the StreamSets software package

I would like to download the StreamSets software package, but when I accesshttps://archives.streamsets.com/index.html, it shows Error 405 Not allowed. How can I resolve this issue? 

0
S
Fan
20 days ago
N
Fan
NateFan
asked in Community Articles and Got a Question?

Python SDK - How to detect new commits pt2.

Hello, I had a post yesterday: Python SDK - How to detect new commits. | StreamSets CommunityHowever, I cannot find how to reply with another question to the response I got so I will post it here in a new question: Just to clarify what I am asking; I would like to see all new commits across all pipelines. So, say in the last day, 2 existing pipelines have new commits, and one new pipeline is created. I would like to be able to see a log that reports there 3 pipelines have had new commits made, and what their ID /commit_id is. Would the SDK happen to support some sort of logging mechanism like this? I was not able to find it in the docs.Thank you,-Nate

0
N
Fan
21 days ago
N
Fan
nik_shahFan
asked in Community Articles and Got a Question?

Writing value in Salesforce Master Detail Relationship field

I have a scenario where I need to populate the value of a parent object on its child record which is in a master detail relationship. For instance, I have a Custom Object “Customer Number” which is in Master Detail relationship with Leads. Instead of pass Salesforce ID, I want to pass the Lead.Name (as it is available in Streamsets) to establish the relationship of “Customer Number” record to its parent Lead object. What is the syntax that I need to use?

0
N
Fan
21 days ago
N
Fan
NateFan
asked in Community Articles and Got a Question?

Python SDK - How to detect new commits.

Hello,  I have been looking through the SDK docs looking for a functionality that would let me see all new pipeline commits made in a certain time frame. I wasn’t having much luck finding anything and I was wondering if there are any functions that support this. Thank you, -Nate

2
N
Fan
22 days ago

Badge winners

Show all badges
Terms & Conditions