Community Articles and Got a Question?

Can't find what you're looking for? Ask it here or check out the Community articles

  • 902 Topics
  • 2,157 Replies

902 Topics

southwr62
Fan
southwr62Fan
asked in Community Articles and Got a Question?

Problem with Pagination

Hi, I am using the HTTP client origin, and since I need to load a lot of records, I need to use pagination.my url ends with ‘auditLogs?offset=${startAt}&pagesize=5&isascorder=false&order=id’ cannot give more for confidentiallity reasons. This is a post requestwhen I set pagination to offset or page number, all I get is the first 5 records, offset is actually used to denote the start record of the data returned, not a page in this instance.If I set pagesize to 1, which I think is how I need to do this, I only get 1 result it doesn’t seem to continue, I have tried with both page and offset set as the pagination method.any ideas on what is wrong are welcomed

0
southwr62
Fan
2 days ago
Sperch
Fan
SperchFan
asked in Community Articles and Got a Question?

Troubleshooting Jython Evaluator Code

I am trying to use a Jython evaluator stage to remove the formatting and code encapsulation to only present the string content to be inserted in the destination stage (Snowflake). I have been searching online for ideas and guidance but have found very little, and most not helpful and/or pertinent to my quandary. An example of the string incoming to the Jython stage:<p style="margin: 0in 0in 0pt;"><span style="color: #000000; font-family: verdana, geneva; font-size: 10pt;">This position is embedded within our Data Analytics practice. Leverage your curiosity and problem-solving skills to explore, discover, and predict patterns contained within data sets for a wide range of government clients. This includes the derivation of clear narratives that help our clients understand their data and how those insights address their research questions.</span></p><p style="margin: 0in 0in 0pt;"> </p><p style="margin: 0in 0in 0pt;"><span style="color: #00000

4
Sperch
Fan
3 days ago
southwr62
Fan
southwr62Fan
asked in Community Articles and Got a Question?

Calculating a Parameter

Hi,I need to use a calculated date in a pipeline, more specifically I need to make an HTTP call, where a date range of the form of a ‘From’ and ‘To’ date, of the form “2023-09-10”.I can calculate this date as a field in a record using  ${time:extractStringFromDate(time:dateAddition("DAY",-1,time:now()),"yyyy-MM-dd")}  But this does not work in the ‘required data’, what would normally be the request body in a post requestIt seems I can use a parameter in this section as follows {"entity":"user","operation":"Login","userName":null,"fromDate":${FDate},"toDate":${TDate},"toDateVal":"2022-07-21T00:00:00.000Z","fromDateVal":"2022-01-01T00:00:00.000Z"}but when I try to create a parameter that calculates the date, using the above formula, I get an EL expression error Can someone suggest where I am going wrong?I have also tried using start jobs, to pass the dates as a parameter, but that also doesn’t seem to work, I can pass a manually entered date to this ‘child job’ but it seems to conv

1
Bikram
Headliner
4 days ago
S
Fan
shobhit_pathakFan
asked in Community Articles and Got a Question?

DB2 CDC Pipeline

I want to create a CDC pipeline for DB2 Database, as there is no origin present for the implementation,Is there any alternative to perform CDC for this Database?

8
Kartik Medavarapu
Roadie
4 days ago
N
Opening Band
N_aliOpening Band
asked in Community Articles and Got a Question?

504 error - Gateway Timeout.

 i have here an sftp origin that i want to pull the files from do some formatting on it and then push it as a bulk post request to HTTP client destination.i keep getting this error 504 gateway timeout with the files with size 200KB or bigger.it is posting the data on the system but returns this error. i tried setting the read timeout configuration and maximum time out values to a much higher values like 15 mins or so in the HTTP client destination but i keep getting the same error though the data is posted in the destination.what should i do to prevent this error?

0
N
Opening Band
4 days ago
saleempothiwala
Headliner
saleempothiwalaHeadliner
asked in Community Articles and Got a Question?

Record Header Attributes for Oracle Bulkload

I am trying to use Oracle Bulkload Origin to read multiple tables from Oracle database. I want to use the tablenames from source to write to destination as files names. In the preview I can see some attributes but when I try to capture them in Expression Evaluator they appear null.  Here is my config for expression evaluator Is it something to do with oracle version? or data collector version?I am using Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - ProductionVersion 21.3.0.0.0SDC version 5.5.0 I can see the jdbc.tables if i use JDBC Multitable Consumer but I want to use Bulkload.

3
saleempothiwala
Headliner
5 days ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in Community Articles and Got a Question?

how to convert data type of decimal into datetime .

this is my datetime values comes from xlsx file and i want to convert into proper datetime VENDOR_CREATION_DATE COLUMN AS A  decimal data type --->want to change datetime I used field type convertor :-to achieve this but its not working ,any one can provide the configuration   

2
Sperch
Fan
5 days ago
southwr62
Fan
southwr62Fan
asked in Community Articles and Got a Question?

Start Jobs processor not passing string as a string

I am trying to create a master flow that calls another pipeline but am having issues in that when I pass parameters, they seem to be converted to values, or at least they lose the quote marks {      "TDate": "2023-09-17",      "FDate": "2023-09-14"}In the status window of the job called, they show up as  2023-09-17 and 2023-09-14, instead of “2023-09-17” and “2023-09-14”, and also update the job parameters without the quotes  If I try and calculate the dates {      "TDate": "${record:value('/TDateY')}",      "FDate": "${record:value('/FDateY')}"}I get the following errorSTART_JOB_05 - Failed to parse runtime parameters for job ID: START_JOB_05 - Failed to parse runtime parameters for job ID: 6f86b611-bfbe-4882-a47f-3aa41f0daeab:0f246ae0-4677-11ec-a298-93161f808487, error: com.fasterxml.jackson.core.JsonParseException: Unexpected character (' ' (code 160)): was expecting double-quote to start field name at [Source: (String)"{       "TDate": "2023-09-17",       "FDate": "2023-09-17" }";

1
Bikram
Headliner
6 days ago
S
Fan
skyFan
asked in Community Articles and Got a Question?

StreamSets Full RPM package download

I would like to download StreamSets Full RPM package of version 3.14.0, but I am unable to do so through the official website. Could someone please send me a copy? I would be very grateful. My email address is 1028259318@qq.com. Thank you in advance.

2
S
Fan
7 days ago
S
Fan
sgrewalFan
asked in Community Articles and Got a Question?

Excel File as Origin Source using HTTPS

I have a bunch of excel (xls) files that are hosted online (HTTPS) and periodically updated. I want to be able to store the excel file to ADLS Gen2. What is the best way to approach this? Data SourceNo authentication/authorization needed. HTTPS XLS formatEnvironment StackStreamsets Azure Shop

14
Bikram
Headliner
7 days ago
M
Fan
Mihir KaleFan
asked in Community Articles and Got a Question?

Reading stage libraries from SDK

Requirement: I want to create an upgraded version of new deployment from an old deployment using SDK. The 2 deployments may have different set of stage libraries depending on their engine versions. So, I want to fetch list of all libraries for each version, compare the old deployment’s libraries with that of the obtained repository comprising upgraded version’s libraries, to check for any deprecated/upgraded versions, and then create the new deployment with the updated stage library list. I want to access all possible stage libraries (common + enterprise) of a particular engine version using SDK. Is there an API to fetch this?

0
M
Fan
9 days ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in Community Articles and Got a Question?

insert blank values in xlxs file into a Null Values ?

i used expression evaluator to give or insert blank record values into Null Values ,how can i achieve this .this is my incoming data from origin. these type field had Blank ,but i want to feed Null ..i used this expression:${str:isNullOrEmpty(record:value('/TYPE_1099'))?NULL:record:value('/TYPE_1099')} but it fails . 

11
lakshmi_narayanan_t
Discovered Fame
11 days ago
Anmuthu
Fan
AnmuthuFan
asked in Community Articles and Got a Question?

Generating blank output files

I have a Kafka topic as a source and the sink as a delimited files. I want to generate a blank output file if there is no data in the topic. What is the best way to achieve this.  thanks

3
Bikram
Headliner
11 days ago
Y
Discovered Fame
yogesh0590
asked in Community Articles and Got a Question?

How to get header (all colums) field as new column ?

Hi Team,I have scenario where I am reading delimited file from S3, in next step I need to compare header of this file. I am unable to get header as new field along with other attributes. @Bikram @Ranjith P @Rishi Any idea on this ? TIA 

3
Bikram
Headliner
11 days ago
Y
Discovered Fame
yogesh0590
asked in Community Articles and Got a Question?

Join operation in SDC

Hi Team,I need to perform a join operation in StreamSets Control Hub using SDC(Data Collectors). As I checked we don’t have any processor or stage in StreamSets SDC.  Is there any workaround we have in SDC. Below are my datasets,Dataset 1 - CSVDataset 2 - Snowflake table Any way to join Dataset 1 and Dataset 2 in SDC. TIA@Rishi @Bikram  @Maria Vila @XavierV  @AkshayJadhav 

3
Bikram
Headliner
11 days ago
M
Fan
Mihir KaleFan
asked in Community Articles and Got a Question?

Incorrect job metrics in orchestration pipeline

I want to evaluate the job metrics field of orchestrator tasks JSON, which is the output of start job origin in orchestration pipeline. The start job origin contains pipeline which moves 5 records from csv file from directory to snowflake table.I need to confirm from the job metrics field that the pipeline output count equals that of input count.But when I preview the orchestration pipeline, both the input and output records count show 0.Is there any additional configuration required that I have missed? I have attached the preview SS along with the job instance pipeline.

2
prasanna_kumar
Roadie
11 days ago
P
Discovered Fame
Priyanka MynepallyDiscovered Fame
asked in Community Articles and Got a Question?

File upload to HTTP

I have a file in resources directory.I need to upload whole file through HTTP.What options to cnfigure. Able to do it in postman

3
Y
Fan
12 days ago
S
Fan
skyFan
asked in Community Articles and Got a Question?

download the StreamSets software package

I would like to download the StreamSets software package, but when I accesshttps://archives.streamsets.com/index.html, it shows Error 405 Not allowed. How can I resolve this issue? 

0
S
Fan
12 days ago
N
Fan
NateFan
asked in Community Articles and Got a Question?

Python SDK - How to detect new commits pt2.

Hello, I had a post yesterday: Python SDK - How to detect new commits. | StreamSets CommunityHowever, I cannot find how to reply with another question to the response I got so I will post it here in a new question: Just to clarify what I am asking; I would like to see all new commits across all pipelines. So, say in the last day, 2 existing pipelines have new commits, and one new pipeline is created. I would like to be able to see a log that reports there 3 pipelines have had new commits made, and what their ID /commit_id is. Would the SDK happen to support some sort of logging mechanism like this? I was not able to find it in the docs.Thank you,-Nate

0
N
Fan
12 days ago
N
Fan
nik_shahFan
asked in Community Articles and Got a Question?

Writing value in Salesforce Master Detail Relationship field

I have a scenario where I need to populate the value of a parent object on its child record which is in a master detail relationship. For instance, I have a Custom Object “Customer Number” which is in Master Detail relationship with Leads. Instead of pass Salesforce ID, I want to pass the Lead.Name (as it is available in Streamsets) to establish the relationship of “Customer Number” record to its parent Lead object. What is the syntax that I need to use?

0
N
Fan
12 days ago
N
Fan
NateFan
asked in Community Articles and Got a Question?

Python SDK - How to detect new commits.

Hello,  I have been looking through the SDK docs looking for a functionality that would let me see all new pipeline commits made in a certain time frame. I wasn’t having much luck finding anything and I was wondering if there are any functions that support this. Thank you, -Nate

2
N
Fan
13 days ago
Douglas R
Fan
Douglas RFan
posted in Community Articles and Got a Question?

"Failed at Step EXEC - Permission denied" when starting SDC as a Service on Systems with SELinux

"Failed at step EXEC  - Permission denied" when starting SDC as a Service on Systems with SELinux ProblemWhen starting StreamSets Data Collector as a service under systemd, the service fails immediately on startup.  The following error is shown when the status of the service is checked (using the systemctl status sdc command).systemd[13801]: sdc.service: Failed at step EXEC spawning /opt/streamsets-datacollector/bin/streamsets: Permission denied CauseThis error indicates that the script SDC_HOME/bin/streamsets (which is used to start SDC) could not be launched by systemd.   This can be caused by incorrect file or directory permissions or by problems with the SELinux context (on systems where SELinux is enabled). SolutionStep 1:  Examine the Service ConfigurationBefore looking at possible permissions issues, it is important to verify that the user, group, and installation directory for the service are correctly configured in the service unit file, /etc/systemd/system/sdc.service.  Exami

0
Douglas R
Fan
16 days ago
Douglas R
Fan
Douglas RFan
posted in Community Articles and Got a Question?

MySQL: JDBC_06: Failed to Initialize Connection Pool: Access Denied Error

MySQL: JDBC_06: Failed to Initialize Connection Pool: Access Denied Error ProblemA pipeline which is using a JDBC stage to access a MySQL database generates an error int he following form:START_ERROR: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Access denied for user 'username'@'hostname' (using password: YES)The pipeline is using a configured Connection in Control Hub, and when the Test Connection button is used in the Edit Connection window, the connection succeeds.  Only when it is used in an actual pipeline does it fail.CauseThis behavior occurs when two conditions are met:MySQL is configured to require SSL.  This condition can be met in one of three ways: The MySQL user is configured with the REQUIRE SSL option. The “requireSSL” property has been configured in the JDBC Additional Configuration Properties int he JDBC stage, with the value “true” “requireSSL=true” has been added to the JD

0
Douglas R
Fan
16 days ago
southwr62
Fan
southwr62Fan
asked in Community Articles and Got a Question?

How to use time:dateAddition

Hi, as a relatvie newcomer to Streamsets, I am trying to use ‘time:dateAddition’ function, in data collector. I am using an engine version 5.6.1, but get the same issues with 5.5.0I am basically trying to use time:Now(), to calculate yesterdays date (or at least a different date), but getting nowhere.I even tried using the example in the docs “${time:dateAddition("MONTH", 2, 2007-10-03T10:15:30)}”but get the error‘EXPR_00 - Invalid expression '${time:dateAddition("MONTH", 2, 2007-10-03T10:15:30)}': com.streamsets.pipeline.api.el.ELEvalException: CTRCMN_0101 - Error parsing expression ${time:dateAddition("MONTH", 2, 2007-10-03T10:15:30)}: javax.servlet.jsp.el.ELException: Encountered "T10", expected one of [".", ">", "gt", "<", "lt", "==", "eq", "<=", "le", ">=", "ge", "!=", "ne", ")", ",", "[", "+", "-", "*", "/", "div", "%", "mod", "and", "&&", "or", "||", "?"]’This function seems to be new with 5.5.0Could someone help please.

4
Bikram
Headliner
17 days ago
S
Fan
santhoshchallaFan
asked in Community Articles and Got a Question?

Find the status of a job using REST API

Hi,I have triggered one of the job suing REST API in StreamSets and I need to the status of the job if it is failed or successful. I am able to get the status of the job but if the job is failed or successful, I am getting status as Inactive in both success and failure case. I need to get exact status like success or failure and error message in case of job failure. Please let me know how I can I achieve this StreamSets. 

6
prasanna_kumar
Roadie
17 days ago

Badge winners

Show all badges
Terms & Conditions