Skip to main content

StreamSets Platform

Get inspired and gain all the knowledge you need

1190 Topics
S
Headliner
Srinivasan SankarHeadliner
asked in Community Articles and Got a Question?
Looping in Data Collector pipelines

Hi there, Any design suggestions to build a loop in StreamSets Data Collector pipelines ?  Example:I have two API endpoints, the first one takes in a request to perform an activity and sends me an ID for the request placed. As a second step, I need to poll a different endpoint with the ID obtained from the previous step to check the status of my earlier request (i.e. placed in the first step). Ideally, I would like to loop that second endpoint in my StreamSets pipeline until I get the status as “succeeded” or “error”. Any suggestions on how I achieve this in a data collector pipeline? Cheers,Srini

2
prasanna_kumar
Roadie
1 year ago
msc
Fan
mscFan
asked in Community Articles and Got a Question?
SOOLDIR_01 & SPOOLDIR_35 in Hadoop FS standalone

Hi,I’m using Hadoop FS standalone as origin in the pipeline. With read order last modified timestampThe file directory is : /user/msc/*           File name pattern is : *Under msc there are multiple folders and in these folders the hadoop will read all the files present, some functions are ran on those file and move the files to some other locations. The pipeline is working fine but sometimes I get a error like SPOOLDIR_01- failed to process file. Even though the file is read and processed , I’m getting this error.I’m also getting a error like Running error : SPOOLDIR_35- spool directory runner failed reason java.io.Filenotfoundexception: file does not exist. After this the pipeline restarts itself. Please help me out, if anyone know the reason. Thanks,Madhusudan

9
Sanjeev
StreamSets Employee
1 year ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in Community Articles and Got a Question?
any performance issue or Lag happens ?

as i running my pipeline continuously ,i scheduled to run pipelines every working day  due to everyday run there is a data generating daily ,like logs ,pipeline schedule history and so on data is generating in my data collector ,already my data collector is running on minimal resources ,due to daily generating data can cause any future performance lag or not.. If yes means what are steps required to prevent performance lag.

1
Sanjeev
StreamSets Employee
1 year ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in Community Articles and Got a Question?
Streamset provide any Reporting tool?

If any reporting tool provided by streamsets.

1
Sanjeev
StreamSets Employee
1 year ago
A
Fan
anand_rajan
asked in Community Articles and Got a Question?
To extract the latest file from the SharePoint Library

Hi,I have a SharePoint site that has csv files in the document library pages. I need to extract the latest modified file for each run based on the last run datetime. So, for example, my last run datetime is '2023-07-20T22:50:10Z', then I need to get the file that has been modified after that time and also the @microsoft.graph.downloadUrl. I use HTTPClient source in Streamsets and use the below as Resource URL. Though it doesn't show any error, I could not get the latest file that has been modified.Please could you help me to know how to get the @microsoft.graph.downloadUrl and the latest file that has been modified using httpclient? https://graph.microsoft.com/v1.0/sites/company.sharepoint.com,<id>/drives/{driveid}/root/children?select=name,fileSystemInfo,@microsoft.graph.downloadUrl&jql=${str:urlEncode(str:concat(str:concat("fileSystemInfo/lastModifiedDateTime ge'",'2023-07-20T22:50:10Z'),"'"), 'UTF8')}In the below example, the lastModifiedDateTime": "2023-07-21T00:00:00Z sh

1
Sanjeev
StreamSets Employee
1 year ago
p_carm
Fan
p_carmFan
asked in Community Articles and Got a Question?
Transfomer Kafka origin consumer group

I’m used to configuring the Data Collector Kafka origin with a specific consumer group.  I need this to control Kafka offsets and the Kafka broker requires this.In Transformer, I don’t see any way to define the consumer group.  How is this done ?

3
Sanjeev
StreamSets Employee
1 year ago
A
Fan
anand_rajan
asked in Community Articles and Got a Question?
timestamp tz to ntz utc conversion.

Hi, I have a record value in the format 2023-07-20T22:50:10Z utc time zone. case 1: I need to convert that to NTZ and add 8 hours to it to compare with the last loaded NTZ timestamp that is UTC+8 hours. Please case2: I need to preserve the same by only removing TZ format. Please could someone provide me help. Thanks. 

6
Bikram
Headliner
1 year ago
H
Fan
HaripriyaFan
asked in Community Articles and Got a Question?
Need to setup notification when job count is Zero records after 10 mins of starting time

Need to add send email after 10 mins of job run when input is zero.

2
Bikram
Headliner
1 year ago
lex03
Discovered Fame
lex03Discovered Fame
asked in Community Articles and Got a Question?
Reading a pdf

I have a pdf on my local system and I want to read the text from it.So I put directory as the origin and selected data format as the whole file and used a jython processor to read the file but it failed.Can someone help me create a pipeline to read the text from the pdf...?

6
lex03
Discovered Fame
1 year ago
S
Fan
SELINJFan
posted in Community Articles and Got a Question?
Need to receive alerts when input is zero.

I'm using oracle CDC pipeline, Sometimes it is not fetching any data, so in that cases I need to get notified via mail.(Ie: We triggered the job , and job is running. But even after if 15 minutes the input is still zero. In this scenario I should get mail ).Is there any way to get notified like this, If it is  either directly from streamsets UI or through Databricks it is acceptable?

1
Sanjeev
StreamSets Employee
1 year ago
S
Discovered Fame
saikat_chatterjeeDiscovered Fame
asked in Community Articles and Got a Question?
How to use mover tool to migrate data from one on-prem dc to another on-prem dc

I have used mover tool commands to migrate data like pipelines, jobs, users etc from one older version SCH (say 3.X) to a new one (say DataOps).Below are the commands used for migration: a) export SCH_USER_ID=<user_id>b) export SCH_PASSWORD=<password>c) export WORK_DIR=<Local system file path>d) export SCH_URL=<url> set URL of the SCH e) export STREAMSETS_SDK_ACTIVATION_KEY='activation key' sudo docker run -e "STREAMSETS_SDK_ACTIVATION_KEY=${STREAMSETS_SDK_ACTIVATION_KEY}" -it -v ${WORK_DIR}:/out streamsets/mover_tool -v export --sch-server-url ${SCH_URL} --sch-user-id "${SCH_USER_ID}" --sch-password "${SCH_PASSWORD}" Now I want to do the same migration from one on-prem sdc to another sdc.Is this possible? If yes can someone help me in this. 

5
Sanjeev
StreamSets Employee
1 year ago
A
StreamSets Employee
ajinkyaStreamSets Employee
asked in Community Articles and Got a Question?
JDBC_06 - Failed to initialize pool: Communications link failure

Issue:While using JDBC stages pipelines are failing with following error:-JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Communications link failureCaused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)Resolution:add ?enabledTLSProtocols=TLSv1.2 in the JDBC URL. Cause:The source and destination targets couldn’t agree on the cipher suites or the protocol to use for the ssl communication. 

0
A
StreamSets Employee
1 year ago
A
StreamSets Employee
ajinkyaStreamSets Employee
asked in Community Articles and Got a Question?
Why preview in Transformer is slow compared to Data Collector?

Preview in StreamSets Transformer is expected to be slow compared to StreamSets Data Collector as Transformer submits the actual job to the spark and fetch the values without writing them to the destination. Review the documentation for the more options.  

0
A
StreamSets Employee
1 year ago
A
StreamSets Employee
ajinkyaStreamSets Employee
asked in Share your Best Practices
How to tune Oracle CDC parameters to get best performance?

Following parameters can be tuned to get the optimal performance from the Oracle CDC:-Max Batch Size (records) = 20000 Batch Wait Time (secs) = 15 - 30 seconds  JDBC Fetch Size for Current Window = 25000 at least JDBC Fetch Size for Past Windows = 25000 at least Time between Session Windows (ms) = 1 ms Time after Session Window Start (ms) = 1ms Parsing Thread Pool Size = 8 (up to 2x the number of CPUs available) NOTE:- Performance depends on the ENV, Infrastructure, Resources, so the values for every environment/use cases may differ. Above mentioned values can be used as a starting point. Tuning performance is trial and error method. Once the optimal values found for the specific environment/use case, those values can be applied. 

0
A
StreamSets Employee
1 year ago
xarnthehero
Fan
xarntheheroFan
asked in Community Articles and Got a Question?
How to alert on error records once a day

This is my first time working with the alerting rules and I’m struggling to figure out if it is possible to create an alert that will trigger multiple times when a threshold of error messages is reached, resetting after each alert. For example, my job typically has no error records, but recently in our test environment we have been receiving some due to bad input data. I’d like to build an alert that sends me an email once a day if any error records were received that day. I’m trying to avoid getting an alert for every batch of records that has an error record, especially if this alert will go into the production job. Reading over the Alerts and Rules documentation, none of the alert types seem like they fulfill this case. I can get an alert as soon as I get any error records, but I don’t see a way to reset it after a time period except for looking at each batch of records. Any help here would be appreciated!

2
Sanjeev
StreamSets Employee
1 year ago
himanshu1234567
Discovered Fame
himanshu1234567Discovered Fame
asked in Community Articles and Got a Question?
How to read files from s3 bucket using groovy?

Can someone provide working code of groovy to read files from s3 bucket?

7
Sanjeev
StreamSets Employee
1 year ago
P
Discovered Fame
Priyanka MynepallyDiscovered Fame
asked in Community Articles and Got a Question?
Need a cron expression

Hi TeamCan you plz share me cron expression which runs monthly once in first of JAN, APR, JULY, OCT at 2 am CET Thanks in advance RegardsPriyanka. M 

2
P
Discovered Fame
1 year ago
HariUmesh
Fan
HariUmeshFan
asked in Community Articles and Got a Question?
Fetching pipeline names using python sdk takes long time than expected ?

I am trying to get all pipeline name with below code statement from streamsets.sdk import Controlhubsch = Controlhub(credential_id="my_id", token ="Mytoken", use_webscoke_tunneling=False)pipelines = sch.pipeline.namefilterd_name = [name for name in pipelines if name.startswith('abc')] i observed its taking more than an minute to fetch all pipeline names in to the list Is the righ approach to filter pipeline names from control hub or is there any issue on streamset sdk?  

1
dima
StreamSets Employee
1 year ago
som
Fan
somFan
posted in Community Articles and Got a Question?
How to pass json file as parameter in RESTAPI call

ISSUE:We would like to pass json file as a parameter in the REST API Call to update an ACL.API I want to use is - /jobrunner/rest/v1/sdc/{sdcId}/acl.Solution: 1- Login to SCH security app:curl -X POST -d '{"userName":"DPMUserID", "password": "DPMUserPassword"}' https://cloud.streamsets.com/security/public-rest/v1/authentication/login --header "Content-Type:application/json" --header "X-Requested-By:SDC" -c cookie.txt2. Generate auth token from the security app:sessionToken=$(cat cookie.txt | grep SSO | rev | grep -o '^\S*' | rev)echo "Generated session token : $sessionToken"After that you can call the SCH REST API using the auth token: Command to fetch the Details of acl before updating:(Please change the sdc ID as per your sdc ID)curl -X GET https://cloud.streamsets.com/jobrunner/rest/v1/sdc/<sdc_id>/acl --header "Content-Type:application/json" --header "X-Requested-By:SCH" --header "X-SS-REST-CALL:true" --header "X-SS-User-Auth-Token:$sessionToken" -i Update command:curl -X PO

0
som
Fan
1 year ago
himanshu1234567
Discovered Fame
himanshu1234567Discovered Fame
asked in Community Articles and Got a Question?
Reading json file from aws bucket using groovy script in groovy_scripting.

Hi Team,We are trying to read json file from aws s3 bucket using groovy script in groovy_scripting origin but we are facing one error “RETRY: SCRIPTING_10 - Script error in user script: javax.script.ScriptException: com.amazonaws.SdkClientException: Unable to load partition metadata from com/amazonaws/partitions/endpoints.json “does anyone knows why we are getting this error? code snippet:-  

3
himanshu1234567
Discovered Fame
1 year ago
HariUmesh
Fan
HariUmeshFan
asked in StreamSets Academy
How to get the list of pipeline names from control hub using python sdk ?

Hi All,i am building a unittest framework for validating the pipeline names from the streamsets control hubFor that i need to fetch pipeline names from streamsets using the python platform sdk as seen belowfrom streamsets.sdk import Controlhubsch = Controlhub(credential_id="my_id", token ="Mytoken", use_webscoke_tunneling=False)pipelines = sch.pipeline.get_allthe above code return list of attributes and name of pipeline with the type of class ‘Method’.  i actually need to get only those pipelines that already deployed into the streamsets through our CI platformso what would be the right approach to filter and get the pipeline names from control hub?I don't see any example specific to this scenarioCan you please help on this? 

2
HariUmesh
Fan
1 year ago
mikewatton
Fan
mikewattonFan
asked in StreamSets Academy
External resource unable to find http://18.118.200.208/wp-content/uploads/2021/11/externalResources.zip

Hi Streamsets Team,I was using Streamsets Tutorial - Build a CDC Pipeline, Where its mentioned to use External Resource but unable to find, because this hyperlink does not map correctly.Could you please check and update.   http://18.118.200.208/wp-content/uploads/2021/11/externalResources.zip

0
mikewatton
Fan
1 year ago
himanshu1234567
Discovered Fame
himanshu1234567Discovered Fame
asked in Community Articles and Got a Question?
using groovy_scripting reading data from aws s3 bucket.

Hi Team,We are working on groovy script to connect to s3 bucket and read json file and convert data into stramsets records. We are able to make connection and convert json file into stream object but we are not able to convert stream object into stramsets records. Pls help us.I am providing code snippet. 

0
himanshu1234567
Discovered Fame
1 year ago
H
Fan
harshit_agrawal1Fan
asked in Community Articles and Got a Question?
Shell Executer error: "stderr: sudo: unable to initialize policy plugin"! and one more...

Hi I am trying to use custom scripts to fetch the directory tree from the shell executor While running the script I am facing this error:“stderr: sudo: unable to initialize policy plugin”Another error which I am facing is:“error record: SHELL_003 - Script return non-zero value: 1”Though i have explicitly mentioned “exit 0” at the end of the script.I have uncommented the configration in deployemnt:stage.conf_com.streamsets.pipeline.stage.executor.shell.impersonation_mode=CURRENT_USER Is there any proper syntax which I need to use to make it run or is there something that I am missing? 

0
H
Fan
1 year ago
K
Fan
Kar003Fan
asked in Community Articles and Got a Question?
MQ JMS Wait Interval - JMS Consumer Pipeline

JMS Consumer in pipeline is connecting to an IBM MQ server.The number of GETs is very high, around 50,000 GETs in 15 minutes with Valid GETs (Messages consumed) in the low hundreds.This is usually resolved by setting MQGMO_WAIT and interval set to 60 seconds.In JMS this is set in property “receive.timeout”.Does anyone have idea on how to specify this via the JMS properties in UI of streamiest pipeline ?

2
K
Fan
1 year ago

Badge winners

Show all badges
Terms & Conditions