Community Articles and Got a Question?

Can't find what you're looking for? Ask it here or check out the Community articles

  • 901 Topics
  • 2,157 Replies

901 Topics

N
Opening Band
N_aliOpening Band
asked in Community Articles and Got a Question?

504 error - Gateway Timeout.

 i have here an sftp origin that i want to pull the files from do some formatting on it and then push it as a bulk post request to HTTP client destination.i keep getting this error 504 gateway timeout with the files with size 200KB or bigger.it is posting the data on the system but returns this error. i tried setting the read timeout configuration and maximum time out values to a much higher values like 15 mins or so in the HTTP client destination but i keep getting the same error though the data is posted in the destination.what should i do to prevent this error?

0
N
Opening Band
1 day ago
southwr62
Fan
southwr62Fan
asked in Community Articles and Got a Question?

Calculating a Parameter

Hi,I need to use a calculated date in a pipeline, more specifically I need to make an HTTP call, where a date range of the form of a ‘From’ and ‘To’ date, of the form “2023-09-10”.I can calculate this date as a field in a record using  ${time:extractStringFromDate(time:dateAddition("DAY",-1,time:now()),"yyyy-MM-dd")}  But this does not work in the ‘required data’, what would normally be the request body in a post requestIt seems I can use a parameter in this section as follows {"entity":"user","operation":"Login","userName":null,"fromDate":${FDate},"toDate":${TDate},"toDateVal":"2022-07-21T00:00:00.000Z","fromDateVal":"2022-01-01T00:00:00.000Z"}but when I try to create a parameter that calculates the date, using the above formula, I get an EL expression error Can someone suggest where I am going wrong?I have also tried using start jobs, to pass the dates as a parameter, but that also doesn’t seem to work, I can pass a manually entered date to this ‘child job’ but it seems to conv

1
Bikram
Headliner
1 day ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in Community Articles and Got a Question?

how to convert data type of decimal into datetime .

this is my datetime values comes from xlsx file and i want to convert into proper datetime VENDOR_CREATION_DATE COLUMN AS A  decimal data type --->want to change datetime I used field type convertor :-to achieve this but its not working ,any one can provide the configuration   

2
Sperch
Fan
2 days ago
Sperch
Fan
SperchFan
asked in Community Articles and Got a Question?

Troubleshooting Jython Evaluator Code

I am trying to use a Jython evaluator stage to remove the formatting and code encapsulation to only present the string content to be inserted in the destination stage (Snowflake). I have been searching online for ideas and guidance but have found very little, and most not helpful and/or pertinent to my quandary. An example of the string incoming to the Jython stage:<p style="margin: 0in 0in 0pt;"><span style="color: #000000; font-family: verdana, geneva; font-size: 10pt;">This position is embedded within our Data Analytics practice. Leverage your curiosity and problem-solving skills to explore, discover, and predict patterns contained within data sets for a wide range of government clients. This includes the derivation of clear narratives that help our clients understand their data and how those insights address their research questions.</span></p><p style="margin: 0in 0in 0pt;"> </p><p style="margin: 0in 0in 0pt;"><span style="color: #00000

4
Sperch
Fan
9 hours ago
southwr62
Fan
southwr62Fan
asked in Community Articles and Got a Question?

Start Jobs processor not passing string as a string

I am trying to create a master flow that calls another pipeline but am having issues in that when I pass parameters, they seem to be converted to values, or at least they lose the quote marks {      "TDate": "2023-09-17",      "FDate": "2023-09-14"}In the status window of the job called, they show up as  2023-09-17 and 2023-09-14, instead of “2023-09-17” and “2023-09-14”, and also update the job parameters without the quotes  If I try and calculate the dates {      "TDate": "${record:value('/TDateY')}",      "FDate": "${record:value('/FDateY')}"}I get the following errorSTART_JOB_05 - Failed to parse runtime parameters for job ID: START_JOB_05 - Failed to parse runtime parameters for job ID: 6f86b611-bfbe-4882-a47f-3aa41f0daeab:0f246ae0-4677-11ec-a298-93161f808487, error: com.fasterxml.jackson.core.JsonParseException: Unexpected character (' ' (code 160)): was expecting double-quote to start field name at [Source: (String)"{       "TDate": "2023-09-17",       "FDate": "2023-09-17" }";

1
Bikram
Headliner
3 days ago
saleempothiwala
Headliner
saleempothiwalaHeadliner
asked in Community Articles and Got a Question?

Record Header Attributes for Oracle Bulkload

I am trying to use Oracle Bulkload Origin to read multiple tables from Oracle database. I want to use the tablenames from source to write to destination as files names. In the preview I can see some attributes but when I try to capture them in Expression Evaluator they appear null.  Here is my config for expression evaluator Is it something to do with oracle version? or data collector version?I am using Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - ProductionVersion 21.3.0.0.0SDC version 5.5.0 I can see the jdbc.tables if i use JDBC Multitable Consumer but I want to use Bulkload.

3
saleempothiwala
Headliner
2 days ago
M
Fan
Mihir KaleFan
asked in Community Articles and Got a Question?

Reading stage libraries from SDK

Requirement: I want to create an upgraded version of new deployment from an old deployment using SDK. The 2 deployments may have different set of stage libraries depending on their engine versions. So, I want to fetch list of all libraries for each version, compare the old deployment’s libraries with that of the obtained repository comprising upgraded version’s libraries, to check for any deprecated/upgraded versions, and then create the new deployment with the updated stage library list. I want to access all possible stage libraries (common + enterprise) of a particular engine version using SDK. Is there an API to fetch this?

0
M
Fan
6 days ago
S
Fan
skyFan
asked in Community Articles and Got a Question?

StreamSets Full RPM package download

I would like to download StreamSets Full RPM package of version 3.14.0, but I am unable to do so through the official website. Could someone please send me a copy? I would be very grateful. My email address is 1028259318@qq.com. Thank you in advance.

2
S
Fan
3 days ago
S
Fan
skyFan
asked in Community Articles and Got a Question?

download the StreamSets software package

I would like to download the StreamSets software package, but when I accesshttps://archives.streamsets.com/index.html, it shows Error 405 Not allowed. How can I resolve this issue? 

0
S
Fan
9 days ago
Anmuthu
Fan
AnmuthuFan
asked in Community Articles and Got a Question?

Generating blank output files

I have a Kafka topic as a source and the sink as a delimited files. I want to generate a blank output file if there is no data in the topic. What is the best way to achieve this.  thanks

3
Bikram
Headliner
8 days ago
N
Fan
NateFan
asked in Community Articles and Got a Question?

Python SDK - How to detect new commits pt2.

Hello, I had a post yesterday: Python SDK - How to detect new commits. | StreamSets CommunityHowever, I cannot find how to reply with another question to the response I got so I will post it here in a new question: Just to clarify what I am asking; I would like to see all new commits across all pipelines. So, say in the last day, 2 existing pipelines have new commits, and one new pipeline is created. I would like to be able to see a log that reports there 3 pipelines have had new commits made, and what their ID /commit_id is. Would the SDK happen to support some sort of logging mechanism like this? I was not able to find it in the docs.Thank you,-Nate

0
N
Fan
9 days ago
Y
Discovered Fame
yogesh0590
asked in Community Articles and Got a Question?

Join operation in SDC

Hi Team,I need to perform a join operation in StreamSets Control Hub using SDC(Data Collectors). As I checked we don’t have any processor or stage in StreamSets SDC.  Is there any workaround we have in SDC. Below are my datasets,Dataset 1 - CSVDataset 2 - Snowflake table Any way to join Dataset 1 and Dataset 2 in SDC. TIA@Rishi @Bikram  @Maria Vila @XavierV  @AkshayJadhav 

3
Bikram
Headliner
8 days ago
N
Fan
nik_shahFan
asked in Community Articles and Got a Question?

Writing value in Salesforce Master Detail Relationship field

I have a scenario where I need to populate the value of a parent object on its child record which is in a master detail relationship. For instance, I have a Custom Object “Customer Number” which is in Master Detail relationship with Leads. Instead of pass Salesforce ID, I want to pass the Lead.Name (as it is available in Streamsets) to establish the relationship of “Customer Number” record to its parent Lead object. What is the syntax that I need to use?

0
N
Fan
9 days ago
N
Fan
NateFan
asked in Community Articles and Got a Question?

Python SDK - How to detect new commits.

Hello,  I have been looking through the SDK docs looking for a functionality that would let me see all new pipeline commits made in a certain time frame. I wasn’t having much luck finding anything and I was wondering if there are any functions that support this. Thank you, -Nate

2
N
Fan
10 days ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in Community Articles and Got a Question?

insert blank values in xlxs file into a Null Values ?

i used expression evaluator to give or insert blank record values into Null Values ,how can i achieve this .this is my incoming data from origin. these type field had Blank ,but i want to feed Null ..i used this expression:${str:isNullOrEmpty(record:value('/TYPE_1099'))?NULL:record:value('/TYPE_1099')} but it fails . 

11
lakshmi_narayanan_t
Discovered Fame
7 days ago
Y
Discovered Fame
yogesh0590
asked in Community Articles and Got a Question?

How to get header (all colums) field as new column ?

Hi Team,I have scenario where I am reading delimited file from S3, in next step I need to compare header of this file. I am unable to get header as new field along with other attributes. @Bikram @Ranjith P @Rishi Any idea on this ? TIA 

3
Bikram
Headliner
8 days ago
Douglas R
Fan
Douglas RFan
posted in Community Articles and Got a Question?

"Failed at Step EXEC - Permission denied" when starting SDC as a Service on Systems with SELinux

"Failed at step EXEC  - Permission denied" when starting SDC as a Service on Systems with SELinux ProblemWhen starting StreamSets Data Collector as a service under systemd, the service fails immediately on startup.  The following error is shown when the status of the service is checked (using the systemctl status sdc command).systemd[13801]: sdc.service: Failed at step EXEC spawning /opt/streamsets-datacollector/bin/streamsets: Permission denied CauseThis error indicates that the script SDC_HOME/bin/streamsets (which is used to start SDC) could not be launched by systemd.   This can be caused by incorrect file or directory permissions or by problems with the SELinux context (on systems where SELinux is enabled). SolutionStep 1:  Examine the Service ConfigurationBefore looking at possible permissions issues, it is important to verify that the user, group, and installation directory for the service are correctly configured in the service unit file, /etc/systemd/system/sdc.service.  Exami

0
Douglas R
Fan
13 days ago
Douglas R
Fan
Douglas RFan
posted in Community Articles and Got a Question?

MySQL: JDBC_06: Failed to Initialize Connection Pool: Access Denied Error

MySQL: JDBC_06: Failed to Initialize Connection Pool: Access Denied Error ProblemA pipeline which is using a JDBC stage to access a MySQL database generates an error int he following form:START_ERROR: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Access denied for user 'username'@'hostname' (using password: YES)The pipeline is using a configured Connection in Control Hub, and when the Test Connection button is used in the Edit Connection window, the connection succeeds.  Only when it is used in an actual pipeline does it fail.CauseThis behavior occurs when two conditions are met:MySQL is configured to require SSL.  This condition can be met in one of three ways: The MySQL user is configured with the REQUIRE SSL option. The “requireSSL” property has been configured in the JDBC Additional Configuration Properties int he JDBC stage, with the value “true” “requireSSL=true” has been added to the JD

0
Douglas R
Fan
13 days ago
S
Fan
shobhit_pathakFan
asked in Community Articles and Got a Question?

DB2 CDC Pipeline

I want to create a CDC pipeline for DB2 Database, as there is no origin present for the implementation,Is there any alternative to perform CDC for this Database?

8
Kartik Medavarapu
Roadie
1 day ago
A
StreamSets Employee
ajinkyaStreamSets Employee
asked in Community Articles and Got a Question?

How to get a specific pipeline using pipeline id or pipeline name using python SDK?

sch.pipelines.get(pipeline_id='id of pipeline') function can be used to get the specific pipeline using the pipeline id.  from streamsets.sdk import ControlHubserver_url = 'https://cloud.streamsets.com/'sch = ControlHub(server_url='https://cloud.streamsets.com/',username = 'username@org',password = 'password')desiredpip=sch1.pipelines.get(pipeline_id='00a2a5e6-5125-4477-b022-5507c7f63c59:dpmsupport')desiredpip=sch1.pipelines.get(name='SDC500 - Dev to Trash (batch size 10000)')NOTE:- Fetching a pipeline using pipeline id is much faster than the pipeline name.

0
A
StreamSets Employee
16 days ago
J
Opening Band
jyuanOpening Band
asked in Community Articles and Got a Question?

How to get statistics like bytes streamed per job per day?

Is there a way to gather statistics on jobs?I am interested in showing the bytes streamed by each job in the last 24 hours, and the like. Also, if possible, I would like to retrieve the bytes streamed in the last 30 days. From the ControlHub GUI, I don’t see anything. Maybe the solution is available on the SDC side? Maybe we can run a script on the SDC server to gather statistics?Thanks!

0
J
Opening Band
17 days ago
southwr62
Fan
southwr62Fan
asked in Community Articles and Got a Question?

How to use time:dateAddition

Hi, as a relatvie newcomer to Streamsets, I am trying to use ‘time:dateAddition’ function, in data collector. I am using an engine version 5.6.1, but get the same issues with 5.5.0I am basically trying to use time:Now(), to calculate yesterdays date (or at least a different date), but getting nowhere.I even tried using the example in the docs “${time:dateAddition("MONTH", 2, 2007-10-03T10:15:30)}”but get the error‘EXPR_00 - Invalid expression '${time:dateAddition("MONTH", 2, 2007-10-03T10:15:30)}': com.streamsets.pipeline.api.el.ELEvalException: CTRCMN_0101 - Error parsing expression ${time:dateAddition("MONTH", 2, 2007-10-03T10:15:30)}: javax.servlet.jsp.el.ELException: Encountered "T10", expected one of [".", ">", "gt", "<", "lt", "==", "eq", "<=", "le", ">=", "ge", "!=", "ne", ")", ",", "[", "+", "-", "*", "/", "div", "%", "mod", "and", "&&", "or", "||", "?"]’This function seems to be new with 5.5.0Could someone help please.

4
Bikram
Headliner
13 days ago
AdamK
Fan
AdamKFan
asked in Community Articles and Got a Question?

How can I extend default timeout value?

Hi, I’m using StreamsetsSDK in version 3.12.1.In my code which is run on Teamcity agent when I’m publishing changes to pipeline using publish_pipeline() method I’m getting the following error:requests.exceptions.ConnectionError: HTTPSConnectionPool(host='******', port=******): Max retries exceeded with url: /rest/v1/system/info (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f95dca34820>: Failed to establish a new connection: [Errno 110] Connection timed out'))This might be caused by firewall settings because code works fine when run locally.Is there a way to extend timeout limit? On the other hand doing imports using import_jobs() work fine. 

2
AdamK
Fan
16 days ago
som
Fan
somFan
posted in Community Articles and Got a Question?

Control the number of email alerts per job failure.

ISSUE:When the job gets failed, we get many number of email alerts.Below is the subscription set at our end.We have also tried changing the subscription to “max global fail over retries” but it does not give actual error message and gives a generic error,{{"FROM_STATUS":"INACTIVE", "TO_COLOR":"RED", "TO_STATUS":"INACTIVE", "ERROR_MESSAGE":"Job is inactive due to global failover retries exhausted." } }Can we control the number of email alerts with getting actual error message. SOLUTION:We can control the email alert by setting below subscription where we will get only 1-2 email alert when either jobs gives any error or exception and also can capture actual error message.${TO_STATUS=="INACTIVE" && TO_COLOR=="RED" && str:contains(ERROR_MESSAGE,"RUN_ERROR") || str:contains(ERROR_MESSAGE,"exception")}  

0
som
Fan
20 days ago
som
Fan
somFan
posted in Community Articles and Got a Question?

Disable fail over by rest api call in sch.

ISSUE:When we create a job in rest api, by default it creates the job with fail over enabled.Sometimes we need to create the job via rest api  with fail over disabled.we can follow below solution to achieve this.  SOLUTION:While creating the job, we can change the value of field "migrateOffsets" to “false” and it will create the job with fail over disabled.  

0
som
Fan
20 days ago

Badge winners

Show all badges
Terms & Conditions