Skip to main content

Hi,

I am new to Streamsets. 
I have a situation where i have created 3 pipelines on Transformer (Each pipeline runs 3-4 SQLs)  has Business Date as a parameter (Current Date-1).

Use case - I need to trigger the first pipeline and when its completed then the second pipeline should run and when second pipeline is completed then third pipeline should be running. 

I have tried creating an orchestration pipeline on Data collector using start job, created job templates for each pipeline by watching some sample video and reading the documents but unable to achieve the above use case in the orchestration pipeline.

 

Is there any other way to achieve this.

Can anyone please help and advise. I would be obliged for any help received.

Hi,

I am new to Streamsets. 
I have a situation where i have created 3 pipelines on Transformer (Each pipeline runs 3-4 SQLs)  has Business Date as a parameter (Current Date-1).

Use case - I need to trigger the first pipeline and when its completed then the second pipeline should run and when second pipeline is completed then third pipeline should be running. 

I have tried creating an orchestration pipeline on Data collector using start job, created job templates for each pipeline by watching some sample video and reading the documents but unable to achieve the above use case in the orchestration pipeline.

 

Is there any other way to achieve this.

Can anyone please help and advise. I would be obliged for any help received.

 

@Rachit_2022 

First create 3 jobs for each of the pipelines. Make sure they all accept date parameter.

Now create an Orchestration pipeline using DataCollector engine.

Add Expression Evaluator to create relevent date fields using time:now() to get current date and then calculate the required date to be passed as parameters.

Add 3 startjob processors. Each calling your transformer jobs. You can add preconditions on Job2 to check if finishedsuccessfully == true for job 1. And on Job3, add precondition that finisedsuccessfully==true for job2.

 

Hope this helps.

Hi @saleempothiwala ,

Thanks a lot for taking out time to reply to this post.

Do you have any pipeline that you can share with me just for the reference to see how all it is done.

It would be a great help.

Thanks a lot...

@Rachit_2022 

Please find attached the orchestration and do let me know if it helps.

Thanks & Regards

Bikram_ 

@Rachit_2022 

 

please check @Bikram ‘s pipeline to see how you can do orchestration and do conditional execution using StreamSelector.

 

I have attached couple of pipelines too. Orchestration pipeline creates a field called businessdate that is current_date-1. This field is then used to set value of parameter in Job1. Also, you can use Wait for Jobs if you want to wait until the next job finishes.

 

I have set pre-condition on second job to see if first job finishes successfully. Similarly you can add third StartJobs processor and check success for Job 2.

 

You will have to create a job for JDBC pipeline and use the jobID in the Start Jobs processor.

 

I hope it helps.

Hi @Bikram & @saleempothiwala 

Thanks a lot for attaching the pipelines.

Unfortunately, both the attached pipelines are not getting imported on Streamsets as we are getting an error.
 


Request you both to share the pipeline again, so that we can import, open and refer for further work.

Thanks again for such prompt replies and patience.

Reply


Terms & Conditions