StreamSets Academy

can you please suggest any other configuration or proseccor to achieve output.i reading my file in directory origin origin is :configuration my destination is localfs:configuration  after  execution of my pipeline :-out put file like - But my requirement is like  how to acive my fie format ,by using any processor jython evaluvator or whole file transformer.

 Would you like to share How StreamSets Stop or kill jobs running on Data Collector engine? (Control Hub)I ask this because I cannot access the job from the engine detail, in the "Running Pipelines" section. When trying to access the job instance I get the following error: "HTTP Status: 404 (Not Found): "  maybe kill the job from command line?  regards. Carlos.  

We are facing some connection issue from STF side while executing a test code, that might be regarding the organization ID not sure why this issue is occurring it saying for organization the url STF  calling is “api/security/v3/orgs/e4cbaa88-611a-11ed-b310-c96332d26f9a” but url must be ”http://api/security/v3/orgs/e4cbaa88-611a-11ed-b310-c96332d26f9a?”. I’m not sure how to fix this Error. E           requests.exceptions.MissingSchema: Invalid URL 

In STF 3.x we have used "sch_authoring_sdc.get_history(pipeline)"  to get all the information, for 4.x do we have anything like that for accessing the history? as get_history is not working with STF 4.x?

I have a ADLS origin stage where I have a data file. The data file needs to be updated with a different delimiter value in it. After updating the values in the file, I need to overwrite it back to the same file. I am using jython evaluator to do the updates. However, I am getting a new file created in ADLS instead of writing to the existing file.Here is my Jython code that I am using   try: text_attribute = record.value["text"] unicode_text = unicode(text_attribute,'utf-8') modified_text = unicode_text.replace(u'|', u'~') record.value["text"]=modified_text sdc.output.write(record) except Exception as e: sdc.error.write(record, str(e))My Pipeline :ADLS Origin Stage -→ Jython Evaluator -→ ADLS Destination Stage

Hello, I'm utilizing Control Hub 3. x to recreate the customer's environment. Based on the pipeline name (given as a parameter) and the number of days old, I need to clear up old tasks. I made the pipeline shown below, which is also attached.

I WANT TO WRITE ALL THE FIELD VALUES THAT CONTAINS Blank Values INTO NULL I used these expression for to write specific one field = field ${str:length(record:value('/VENDOR_NAME_ALT')) != 0 ? record:value('/VENDOR_NAME_ALT') : 'null'}but i want to use all  fields  that contains blank values means feed with  NULL . any other processor can replace these scenario

Hello,I’m using Control Hub 3.x ( to replicate the customer’s environment) I need to cleanup old jobs based on the pipeline name (passed as parameter) and n days old. I have created the following pipeline: (also attached)  When I run this API call in browser or via this pipeline, I get many jobs which do not match the criteria.Tried also with creator value and some other fields but facing the same problem. Does this API allow filtering the jobs like this? I can change the pipeline to get all jobs first and then add a stream selector to get the jobs I’m interested in, but I’m trying to limit the API response to only what i need. Can someone help?Dhanashri

My original data is in the XLSX spreadsheet file format. However, I would like to save it locally as an XLSX file format, but I am currently unable to do so. Is there an alternative processor or method I can use to achieve this desired file format conversion?

I had Pincode like alpha numeric how can i separate numeric value into specific fieldthis is field value -4900 B ICI RD BUT I WANT Output like -4990 and remainig another field like-B ICI RD  

ERROR LIKEin my directory xml file but not able to fetch records and also i increased max record length-but file length is less than 1mb but i don’t know why unable to fetch my xml data any one can find solution and share it. 

Hi,I am trying to import the pipelines in another user account but, i am facing the below issue.Cannot import pipeline D_AmazonSQS_Origin_JDBC_Dest_Pipeline_CredStr, pipeline with the same commit Id 6f1628b3-d3b4-4d49-9a77-d80fce22717b:13f66132-5e7d-11ed-b52a-23b2c9b2adfb already exists. : PIPELINE_STORE_30I don’t have any pipelines in that user account where I am trying to import.Also, I have created separate predefined connections which is used inside those pipelines. When I am trying to export, it got downloaded with only name and type in connections metadata.json file which is inside pipelines.zip. But credentials details are not there in the json connection file. And also, I cant able to import those connections.json file. Do you know how to import the connections?Can you help me what could be the issue? Thanks,Malathi S

Hi All,There is a plan to read records from DB2 in StreamSets. Can we use JDBC Multitable consumer Origin, if so, is any other additional steps that we need to take care to start reading records?Please confirm the possibilities in StreamSets connecting to DB2. Thank you,

I need to copy the directory or folder that contains multiple types of files, including CSV and PNG, from one GCP bucket and paste it into a folder in another bucket. How can I do this?

Hi Team, We are trying to upgrade the stream set upgrade from 3.16.x to 3.19.X version using the docker container. Queues are not coming up.After doing the analysis, we got to know activation code concept is introduced. Now what is the procedure for getting the activation code using sdc id incase if we don’t have the admin access. Thanks. 

Hello All,we have an usecase where we’ll read the json message from the kafka topic and call the api using the http client stage for the data load. Here, when any stage sees any error we want to send it back to the same source kafka topic from where we are reading the json messages  . But we see the error records are sent to the kafka topic in sdc format and since our pipeline is configured to read json data format it’s failing. Is there any way we can route the error records to same source topic in json format instead of SDC format without creating a new failure topic and streamsets pipeline to handle the error records?

Hi All,i am building a unittest framework for validating the pipeline names from the streamsets control hubFor that i need to fetch pipeline names from streamsets using the python platform sdk as seen belowfrom streamsets.sdk import Controlhubsch = Controlhub(credential_id="my_id", token ="Mytoken", use_webscoke_tunneling=False)pipelines = sch.pipeline.get_allthe above code return list of attributes and name of pipeline with the type of class ‘Method’.  i actually need to get only those pipelines that already deployed into the streamsets through our CI platformso what would be the right approach to filter and get the pipeline names from control hub?I don't see any example specific to this scenarioCan you please help on this? 

Hi Streamsets Team,I was using Streamsets Tutorial - Build a CDC Pipeline, Where its mentioned to use External Resource but unable to find, because this hyperlink does not map correctly.Could you please check and update.   http://18.118.200.208/wp-content/uploads/2021/11/externalResources.zip

 BELOW IS MY SNOWFLAKE ACCOUNT URL ERROR SHOWS    ERROR LIKE THIS: 

i trying to write avro format as wholefile format from kafka to localfs but i got error.below  i share the pipeline ,can you please suggest how to done this 

Hi, I am able to implement HPA (Horizontal Pod Autoscaling) in streamsets by integrating with kubernetes cluster. i can see engines are getting scaled up.  It is possible to do VPA in streamsets?

Hi All,i am looking for Fragment which has been used in Academy Course for CDC Pipeline Demo, they have mentioned they will share with us. But i cant find it anywhere. Can someone help me to share that fragment. Regards,Rohit

I created a pipeline for primary transformers that would run on an EMR cluster. I used the crucial setup information when configuring the cluster.I'm experiencing the following issue:Transformer's pipeline Engine Transformer 

I have created subscription as suggested in related topic but that subscription is not triggered.  I am able to create incident at pipeline level when pipeline fails, using the notification tab by giving webhook details for incident table.Also, I want some information about job failure in incident table for e.g. Short description like job_name is ‘abc’ failed and their error details. Is this possible using subscription? If yes then please help me on the same.

Hi,If the field given in Result Field Path does not exist in the response, then we get an error, which is correct, but how to catch and handle that error to stop the pipeline smoothly.