Runtime parameters: reset origin and start

Question

Hey Community,

I see the option in the Streamsets canvas:

Start pipeline
Reset origin and start
Start with parameters

Is it possible to combine the last 2 options?

Reset origin and start with parameters

icon

Best answer by Srinivasan Sankar 17 December 2021, 11:15

View original

Srinivasan Sankar · Accepted Answer

Hi @ufmurphy,The other option would be,Use a “Pipeline Finisher" executor stage that would reset the origin when the pipeline completes processing. Refer screenshot below.The next time you just click, “Start with Parameters”--Below is an example where a Pipeline Finisher is used in a pipeline. The configuration does not reset the Origin but you could reset the Origin by enabling the check-box.Note - the Pipeline Finisher stops the pipeline and resets the Origin when there are no more data to process from the source. All other events are discarded.Image 1 - Pipeline Finisher Executor Stage used in a Data Collector pipeline.Image 2 - Check the box to reset the origin.Image 3 - Ignore all other events and finish the pipeline ONLY if there is no more data to process from the source.Cheers,

swayam · Answer

Hi @ufmurphyIf you are OK to achieve your requirement via programmatically, then it's trivial to use the SDC REST APIs and achieve the same. You need to call Rest Origin REST API followed by Start pipeline API with runtime variables in POST body.