Skip to main content

StreamSets Academy

StreamSets Academy course topics and questions

  • 107 Topics
  • 254 Replies
107 Topics
S
Roadie
shivrajRoadie
asked in StreamSets Academy
STF

In STF 3.x we have used "sch_authoring_sdc.get_history(pipeline)"  to get all the information, for 4.x do we have anything like that for accessing the history? as get_history is not working with STF 4.x?

452
S
Roadie
1 year ago
R
Fan
rajasriramFan
asked in StreamSets Academy
Read file from ADLS Origin and Update same file in ADLS destination stage

I have a ADLS origin stage where I have a data file. The data file needs to be updated with a different delimiter value in it. After updating the values in the file, I need to overwrite it back to the same file. I am using jython evaluator to do the updates. However, I am getting a new file created in ADLS instead of writing to the existing file.Here is my Jython code that I am using   try: text_attribute = record.value["text"] unicode_text = unicode(text_attribute,'utf-8') modified_text = unicode_text.replace(u'|', u'~') record.value["text"]=modified_text sdc.output.write(record) except Exception as e: sdc.error.write(record, str(e))My Pipeline :ADLS Origin Stage -→ Jython Evaluator -→ ADLS Destination Stage

70
R
Fan
1 year ago
pinkbapehoodie22
Fan
pinkbapehoodie22Fan
asked in StreamSets Academy
3. x - Job runner API - Obtaining jobs for a particular pipeline - returns jobs that do not meet the requirement

Hello, I'm utilizing Control Hub 3. x to recreate the customer's environment. Based on the pipeline name (given as a parameter) and the number of days old, I need to clear up old tasks. I made the pipeline shown below, which is also attached.

181
Rishi
StreamSets Employee
1 year ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in StreamSets Academy
I want to correct the datatypes of all of my input field values ?

I WANT TO WRITE ALL THE FIELD VALUES THAT CONTAINS Blank Values INTO NULL I used these expression for to write specific one field = field ${str:length(record:value('/VENDOR_NAME_ALT')) != 0 ? record:value('/VENDOR_NAME_ALT') : 'null'}but i want to use all  fields  that contains blank values means feed with  NULL . any other processor can replace these scenario

201
Bikram
Headliner
1 year ago
D
Opening Band
Dhanashri_BhateOpening Band
asked in StreamSets Academy
3.x - Job runner API - Getting jobs for a specific pipeline - returns jobs that do not match the criteria

Hello,I’m using Control Hub 3.x ( to replicate the customer’s environment) I need to cleanup old jobs based on the pipeline name (passed as parameter) and n days old. I have created the following pipeline: (also attached)  When I run this API call in browser or via this pipeline, I get many jobs which do not match the criteria.Tried also with creator value and some other fields but facing the same problem. Does this API allow filtering the jobs like this? I can change the pipeline to get all jobs first and then add a stream selector to get the jobs I’m interested in, but I’m trying to limit the API response to only what i need. Can someone help?Dhanashri

220
D
Opening Band
1 year ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in StreamSets Academy
how to Write XLSX file format in localfs.

My original data is in the XLSX spreadsheet file format. However, I would like to save it locally as an XLSX file format, but I am currently unable to do so. Is there an alternative processor or method I can use to achieve this desired file format conversion?

171
Bikram
Headliner
1 year ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in StreamSets Academy
how to separate pincode into specific field?

I had Pincode like alpha numeric how can i separate numeric value into specific fieldthis is field value -4900 B ICI RD BUT I WANT Output like -4990 and remainig another field like-B ICI RD  

181
Rishi
StreamSets Employee
1 year ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in StreamSets Academy
directory not able ready xml file .

ERROR LIKEin my directory xml file but not able to fetch records and also i increased max record length-but file length is less than 1mb but i don’t know why unable to fetch my xml data any one can find solution and share it. 

346
saleempothiwala
Headliner
1 year ago
M
Fan
MalathiFan
asked in StreamSets Academy
Getting error while importing pipeline and cant able to import connections.

Hi,I am trying to import the pipelines in another user account but, i am facing the below issue.Cannot import pipeline D_AmazonSQS_Origin_JDBC_Dest_Pipeline_CredStr, pipeline with the same commit Id 6f1628b3-d3b4-4d49-9a77-d80fce22717b:13f66132-5e7d-11ed-b52a-23b2c9b2adfb already exists. : PIPELINE_STORE_30I don’t have any pipelines in that user account where I am trying to import.Also, I have created separate predefined connections which is used inside those pipelines. When I am trying to export, it got downloaded with only name and type in connections metadata.json file which is inside pipelines.zip. But credentials details are not there in the json connection file. And also, I cant able to import those connections.json file. Do you know how to import the connections?Can you help me what could be the issue? Thanks,Malathi S

661
M
Fan
1 year ago
V
Fan
vigneshmFan
asked in StreamSets Academy
Possibility to connect DB2 as source in StreamSets

Hi All,There is a plan to read records from DB2 in StreamSets. Can we use JDBC Multitable consumer Origin, if so, is any other additional steps that we need to take care to start reading records?Please confirm the possibilities in StreamSets connecting to DB2. Thank you,

942
V
Fan
1 year ago
Vip
Fan
VipFan
asked in StreamSets Academy
Whole file transfer from a GCP bucket to another GCP bucket

I need to copy the directory or folder that contains multiple types of files, including CSV and PNG, from one GCP bucket and paste it into a folder in another bucket. How can I do this?

201
saleempothiwala
Headliner
1 year ago
U
Fan
unifyStreamSet
asked in StreamSets Academy
Stream set upgrade issues with 3.19 version

Hi Team, We are trying to upgrade the stream set upgrade from 3.16.x to 3.19.X version using the docker container. Queues are not coming up.After doing the analysis, we got to know activation code concept is introduced. Now what is the procedure for getting the activation code using sdc id incase if we don’t have the admin access. Thanks. 

333
Sanjeev
StreamSets Employee
1 year ago
nikkichow
Fan
nikkichowFan
asked in StreamSets Academy
Handling the error records in the same Kafka topic

Hello All,we have an usecase where we’ll read the json message from the kafka topic and call the api using the http client stage for the data load. Here, when any stage sees any error we want to send it back to the same source kafka topic from where we are reading the json messages  . But we see the error records are sent to the kafka topic in sdc format and since our pipeline is configured to read json data format it’s failing. Is there any way we can route the error records to same source topic in json format instead of SDC format without creating a new failure topic and streamsets pipeline to handle the error records?

1413
Sanjeev
StreamSets Employee
1 year ago
HariUmesh
Fan
HariUmeshFan
asked in StreamSets Academy
How to get the list of pipeline names from control hub using python sdk ?

Hi All,i am building a unittest framework for validating the pipeline names from the streamsets control hubFor that i need to fetch pipeline names from streamsets using the python platform sdk as seen belowfrom streamsets.sdk import Controlhubsch = Controlhub(credential_id="my_id", token ="Mytoken", use_webscoke_tunneling=False)pipelines = sch.pipeline.get_allthe above code return list of attributes and name of pipeline with the type of class ‘Method’.  i actually need to get only those pipelines that already deployed into the streamsets through our CI platformso what would be the right approach to filter and get the pipeline names from control hub?I don't see any example specific to this scenarioCan you please help on this? 

902
HariUmesh
Fan
1 year ago
mikewatton
Fan
mikewattonFan
asked in StreamSets Academy
External resource unable to find http://18.118.200.208/wp-content/uploads/2021/11/externalResources.zip

Hi Streamsets Team,I was using Streamsets Tutorial - Build a CDC Pipeline, Where its mentioned to use External Resource but unable to find, because this hyperlink does not map correctly.Could you please check and update.   http://18.118.200.208/wp-content/uploads/2021/11/externalResources.zip

30
mikewatton
Fan
1 year ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in StreamSets Academy
HOW TO CONFIGURE SNOWFLAKE ACCOUNT URL

 BELOW IS MY SNOWFLAKE ACCOUNT URL ERROR SHOWS    ERROR LIKE THIS: 

311
K
StreamSets Employee
1 year ago
Praveen P
Fan
Praveen PFan
asked in StreamSets Academy
how to write avro file in wholefile format from kafka to local fs?

i trying to write avro format as wholefile format from kafka to localfs but i got error.below  i share the pipeline ,can you please suggest how to done this 

30413
Praveen P
Fan
1 year ago
R
Fan
RadhaMuthukrishnanMFan
asked in StreamSets Academy
Streamset kubernetes autoscaling : VPA

Hi, I am able to implement HPA (Horizontal Pod Autoscaling) in streamsets by integrating with kubernetes cluster. i can see engines are getting scaled up.  It is possible to do VPA in streamsets?

240
R
Fan
1 year ago
R
Fan
rohit6488Fan
asked in StreamSets Academy
Missing Fragment used in Academy Course for CDC from Mysql to Snowflake

Hi All,i am looking for Fragment which has been used in Academy Course for CDC Pipeline Demo, they have mentioned they will share with us. But i cant find it anywhere. Can someone help me to share that fragment. Regards,Rohit

304
R
Fan
1 year ago
Rhudehoodie52
Fan
Rhudehoodie52Fan
asked in StreamSets Academy
Can the transformer pipeline be configured to run on an AWS EMR cluster?

I created a pipeline for primary transformers that would run on an EMR cluster. I used the crucial setup information when configuring the cluster.I'm experiencing the following issue:Transformer's pipeline Engine Transformer 

00
Rhudehoodie52
Fan
1 year ago
A
Roadie
akshayp2Roadie
asked in StreamSets Academy
Subscriptions created for create incident for failed job in ServiceNow has not triggered.

I have created subscription as suggested in related topic but that subscription is not triggered.  I am able to create incident at pipeline level when pipeline fails, using the notification tab by giving webhook details for incident table.Also, I want some information about job failure in incident table for e.g. Short description like job_name is ‘abc’ failed and their error details. Is this possible using subscription? If yes then please help me on the same.

576
XavierV
StreamSets Employee
1 year ago
A
Discovered Fame
AnkurDiscovered Fame
asked in StreamSets Academy
HTTP Client origin pagination config

Hi,If the field given in Result Field Path does not exist in the response, then we get an error, which is correct, but how to catch and handle that error to stop the pipeline smoothly.  

391
AkshayJadhav
StreamSets Employee
1 year ago
D
Opening Band
Dhanashri_BhateOpening Band
asked in StreamSets Academy
Need more info on the two events for SFTP destination?

Hello,I am referring to this documentation : https://docs.streamsets.com/portal/platform-datacollector/latest/datacollector/UserGuide/Destinations/SFTP.html#concept_sgt_m2m_xhbThere are two Event types - Uses one of the following types:file-closed - Generated when the destination closes a file. wholeFileProcessed - Generated when the destination completes streaming a whole file.If I have to add an executor to move the file from this destination then to some other location upon receiving an event, which event I have to use? Are there any more steps between wholeFileProcessed and file-closed? Is there anything more after the event file-closed is triggered? Dhanashri

282
D
Opening Band
1 year ago
A
Discovered Fame
AnkurDiscovered Fame
asked in StreamSets Academy
Metric Rule giving Alert on False Condition.

We have a Metric Alert in a JDBC Producer Input Counter with below Condition:${value() < 35}The above rule with “less than operator” is giving alert in both the conditions True or False. If we change the operator from “less than “< to “greater than”>. This it is behaving as expected.${value() > 35} Please provide the suggestion or resolution for ${value() < 35} to work as expected i.e. only on True condition.        

232
A
Discovered Fame
1 year ago
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in StreamSets Academy
how to get data time format from mongodb to mysql .

the json data in mongodb had  a date format as a  timestamp in value  but during  process only date shown not timestamp show. how to fix below is my mongodb data:- but  inside of data updatedAt column has timestamp value but in streamsets not shown.how to get timestamp value from mongodb origin.

191
AkshayJadhav
StreamSets Employee
1 year ago

Badge winners

Show all badges
Terms & Conditions