Skip to main content

StreamSets Academy

StreamSets Academy course topics and questions

  • 110 Topics
  • 258 Replies
110 Topics
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in StreamSets Academy
how to covert file and write into localfs as a XLSX file Format.

can you please suggest any other configuration or proseccor to achieve output.i reading my file in directory origin origin is :configuration my destination is localfs:configuration  after  execution of my pipeline :-out put file like - But my requirement is like  how to acive my fie format ,by using any processor jython evaluvator or whole file transformer.

412
Bikram
Headliner
ccariman
Fan
ccarimanFan
asked in StreamSets Academy
How StreamSets Stop or kill jobs running on Data Collector engine?

 Would you like to share How StreamSets Stop or kill jobs running on Data Collector engine? (Control Hub)I ask this because I cannot access the job from the engine detail, in the "Running Pipelines" section. When trying to access the job instance I get the following error: "HTTP Status: 404 (Not Found): "  maybe kill the job from command line?  regards. Carlos.  

720
ccariman
Fan
S
Roadie
shivrajRoadie
asked in StreamSets Academy
Facing an error while testing a pipeline on STF 5.x

We are facing some connection issue from STF side while executing a test code, that might be regarding the organization ID not sure why this issue is occurring it saying for organization the url STF  calling is “api/security/v3/orgs/e4cbaa88-611a-11ed-b310-c96332d26f9a” but url must be ”http://api/security/v3/orgs/e4cbaa88-611a-11ed-b310-c96332d26f9a?”. I’m not sure how to fix this Error. E           requests.exceptions.MissingSchema: Invalid URL 

211
S
Roadie
S
Roadie
shivrajRoadie
asked in StreamSets Academy
STF

In STF 3.x we have used "sch_authoring_sdc.get_history(pipeline)"  to get all the information, for 4.x do we have anything like that for accessing the history? as get_history is not working with STF 4.x?

482
S
Roadie
R
Fan
rajasriramFan
asked in StreamSets Academy
Read file from ADLS Origin and Update same file in ADLS destination stage

I have a ADLS origin stage where I have a data file. The data file needs to be updated with a different delimiter value in it. After updating the values in the file, I need to overwrite it back to the same file. I am using jython evaluator to do the updates. However, I am getting a new file created in ADLS instead of writing to the existing file.Here is my Jython code that I am using   try: text_attribute = record.value["text"] unicode_text = unicode(text_attribute,'utf-8') modified_text = unicode_text.replace(u'|', u'~') record.value["text"]=modified_text sdc.output.write(record) except Exception as e: sdc.error.write(record, str(e))My Pipeline :ADLS Origin Stage -→ Jython Evaluator -→ ADLS Destination Stage

140
R
Fan
pinkbapehoodie22
Fan
pinkbapehoodie22Fan
asked in StreamSets Academy
3. x - Job runner API - Obtaining jobs for a particular pipeline - returns jobs that do not meet the requirement

Hello, I'm utilizing Control Hub 3. x to recreate the customer's environment. Based on the pipeline name (given as a parameter) and the number of days old, I need to clear up old tasks. I made the pipeline shown below, which is also attached.

191
Rishi
StreamSets Employee
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in StreamSets Academy
I want to correct the datatypes of all of my input field values ?

I WANT TO WRITE ALL THE FIELD VALUES THAT CONTAINS Blank Values INTO NULL I used these expression for to write specific one field = field ${str:length(record:value('/VENDOR_NAME_ALT')) != 0 ? record:value('/VENDOR_NAME_ALT') : 'null'}but i want to use all  fields  that contains blank values means feed with  NULL . any other processor can replace these scenario

221
Bikram
Headliner
D
Opening Band
Dhanashri_BhateOpening Band
asked in StreamSets Academy
3.x - Job runner API - Getting jobs for a specific pipeline - returns jobs that do not match the criteria

Hello,I’m using Control Hub 3.x ( to replicate the customer’s environment) I need to cleanup old jobs based on the pipeline name (passed as parameter) and n days old. I have created the following pipeline: (also attached)  When I run this API call in browser or via this pipeline, I get many jobs which do not match the criteria.Tried also with creator value and some other fields but facing the same problem. Does this API allow filtering the jobs like this? I can change the pipeline to get all jobs first and then add a stream selector to get the jobs I’m interested in, but I’m trying to limit the API response to only what i need. Can someone help?Dhanashri

310
D
Opening Band
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in StreamSets Academy
how to Write XLSX file format in localfs.

My original data is in the XLSX spreadsheet file format. However, I would like to save it locally as an XLSX file format, but I am currently unable to do so. Is there an alternative processor or method I can use to achieve this desired file format conversion?

201
Bikram
Headliner
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in StreamSets Academy
how to separate pincode into specific field?

I had Pincode like alpha numeric how can i separate numeric value into specific fieldthis is field value -4900 B ICI RD BUT I WANT Output like -4990 and remainig another field like-B ICI RD  

201
Rishi
StreamSets Employee
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in StreamSets Academy
directory not able ready xml file .

ERROR LIKEin my directory xml file but not able to fetch records and also i increased max record length-but file length is less than 1mb but i don’t know why unable to fetch my xml data any one can find solution and share it. 

427
saleempothiwala
Headliner
M
Fan
MalathiFan
asked in StreamSets Academy
Getting error while importing pipeline and cant able to import connections.

Hi,I am trying to import the pipelines in another user account but, i am facing the below issue.Cannot import pipeline D_AmazonSQS_Origin_JDBC_Dest_Pipeline_CredStr, pipeline with the same commit Id 6f1628b3-d3b4-4d49-9a77-d80fce22717b:13f66132-5e7d-11ed-b52a-23b2c9b2adfb already exists. : PIPELINE_STORE_30I don’t have any pipelines in that user account where I am trying to import.Also, I have created separate predefined connections which is used inside those pipelines. When I am trying to export, it got downloaded with only name and type in connections metadata.json file which is inside pipelines.zip. But credentials details are not there in the json connection file. And also, I cant able to import those connections.json file. Do you know how to import the connections?Can you help me what could be the issue? Thanks,Malathi S

881
M
Fan
V
Fan
vigneshmFan
asked in StreamSets Academy
Possibility to connect DB2 as source in StreamSets

Hi All,There is a plan to read records from DB2 in StreamSets. Can we use JDBC Multitable consumer Origin, if so, is any other additional steps that we need to take care to start reading records?Please confirm the possibilities in StreamSets connecting to DB2. Thank you,

1902
V
Fan
Vip
Fan
VipFan
asked in StreamSets Academy
Whole file transfer from a GCP bucket to another GCP bucket

I need to copy the directory or folder that contains multiple types of files, including CSV and PNG, from one GCP bucket and paste it into a folder in another bucket. How can I do this?

231
saleempothiwala
Headliner
U
Fan
unifyStreamSet
asked in StreamSets Academy
Stream set upgrade issues with 3.19 version

Hi Team, We are trying to upgrade the stream set upgrade from 3.16.x to 3.19.X version using the docker container. Queues are not coming up.After doing the analysis, we got to know activation code concept is introduced. Now what is the procedure for getting the activation code using sdc id incase if we don’t have the admin access. Thanks. 

343
Sanjeev
StreamSets Employee
nikkichow
Fan
nikkichowFan
asked in StreamSets Academy
Handling the error records in the same Kafka topic

Hello All,we have an usecase where we’ll read the json message from the kafka topic and call the api using the http client stage for the data load. Here, when any stage sees any error we want to send it back to the same source kafka topic from where we are reading the json messages  . But we see the error records are sent to the kafka topic in sdc format and since our pipeline is configured to read json data format it’s failing. Is there any way we can route the error records to same source topic in json format instead of SDC format without creating a new failure topic and streamsets pipeline to handle the error records?

1653
Sanjeev
StreamSets Employee
HariUmesh
Fan
HariUmeshFan
asked in StreamSets Academy
How to get the list of pipeline names from control hub using python sdk ?

Hi All,i am building a unittest framework for validating the pipeline names from the streamsets control hubFor that i need to fetch pipeline names from streamsets using the python platform sdk as seen belowfrom streamsets.sdk import Controlhubsch = Controlhub(credential_id="my_id", token ="Mytoken", use_webscoke_tunneling=False)pipelines = sch.pipeline.get_allthe above code return list of attributes and name of pipeline with the type of class ‘Method’.  i actually need to get only those pipelines that already deployed into the streamsets through our CI platformso what would be the right approach to filter and get the pipeline names from control hub?I don't see any example specific to this scenarioCan you please help on this? 

1072
HariUmesh
Fan
mikewatton
Fan
mikewattonFan
asked in StreamSets Academy
External resource unable to find http://18.118.200.208/wp-content/uploads/2021/11/externalResources.zip

Hi Streamsets Team,I was using Streamsets Tutorial - Build a CDC Pipeline, Where its mentioned to use External Resource but unable to find, because this hyperlink does not map correctly.Could you please check and update.   http://18.118.200.208/wp-content/uploads/2021/11/externalResources.zip

40
mikewatton
Fan
lakshmi_narayanan_t
Discovered Fame
lakshmi_narayanan_tDiscovered Fame
asked in StreamSets Academy
HOW TO CONFIGURE SNOWFLAKE ACCOUNT URL

 BELOW IS MY SNOWFLAKE ACCOUNT URL ERROR SHOWS    ERROR LIKE THIS: 

381
K
StreamSets Employee
Praveen P
Fan
Praveen PFan
asked in StreamSets Academy
how to write avro file in wholefile format from kafka to local fs?

i trying to write avro format as wholefile format from kafka to localfs but i got error.below  i share the pipeline ,can you please suggest how to done this 

33413
Praveen P
Fan
R
Fan
RadhaMuthukrishnanMFan
asked in StreamSets Academy
Streamset kubernetes autoscaling : VPA

Hi, I am able to implement HPA (Horizontal Pod Autoscaling) in streamsets by integrating with kubernetes cluster. i can see engines are getting scaled up.  It is possible to do VPA in streamsets?

270
R
Fan
R
Fan
rohit6488Fan
asked in StreamSets Academy
Missing Fragment used in Academy Course for CDC from Mysql to Snowflake

Hi All,i am looking for Fragment which has been used in Academy Course for CDC Pipeline Demo, they have mentioned they will share with us. But i cant find it anywhere. Can someone help me to share that fragment. Regards,Rohit

304
R
Fan
Rhudehoodie52
Fan
Rhudehoodie52Fan
asked in StreamSets Academy
Can the transformer pipeline be configured to run on an AWS EMR cluster?

I created a pipeline for primary transformers that would run on an EMR cluster. I used the crucial setup information when configuring the cluster.I'm experiencing the following issue:Transformer's pipeline Engine Transformer 

00
Rhudehoodie52
Fan
A
Roadie
akshayp2Roadie
asked in StreamSets Academy
Subscriptions created for create incident for failed job in ServiceNow has not triggered.

I have created subscription as suggested in related topic but that subscription is not triggered.  I am able to create incident at pipeline level when pipeline fails, using the notification tab by giving webhook details for incident table.Also, I want some information about job failure in incident table for e.g. Short description like job_name is ‘abc’ failed and their error details. Is this possible using subscription? If yes then please help me on the same.

626
XavierV
StreamSets Employee
A
Discovered Fame
AnkurDiscovered Fame
asked in StreamSets Academy
HTTP Client origin pagination config

Hi,If the field given in Result Field Path does not exist in the response, then we get an error, which is correct, but how to catch and handle that error to stop the pipeline smoothly.  

451
AkshayJadhav
StreamSets Employee

Badge winners

Show all badges
Terms & Conditions