Skip to main content

Hey Community,

 

I see the option in the Streamsets canvas:

  • Start pipeline
  • Reset origin and start
  • Start with parameters

 

Is it possible to combine the last 2 options?

  • Reset origin and start with parameters

Hi @ufmurphy 

If you are OK to achieve your requirement via programmatically, then it's trivial to use the SDC REST APIs and achieve the same. You need to call Rest Origin REST API followed by Start pipeline API with runtime variables in POST body.

Hi @ufmurphy , 

 

The other option would be,

 

  • Use a “Pipeline Finisher" executor stage that would reset the origin when the pipeline completes processing. Refer screenshot below.
  • The next time you just click, “Start with Parameters”

--

Below is an example where a Pipeline Finisher is used in a pipeline. The configuration does not reset  the Origin but you could reset the Origin by enabling the check-box.

 

Note - the Pipeline Finisher stops the pipeline and resets the Origin when there are no more data to process from the source. All other events are discarded. 

Image 1 - Pipeline Finisher Executor Stage used in a Data Collector pipeline.
Image 2 - Check the box to  reset the origin.

 

 

Image 3 - Ignore all other events and finish the pipeline ONLY if there is no more data to process from the source.

 

Cheers,

 

Both of the above answers are excellent. I chose the second as “best” because right now I am working in the UI. But I definitely see the benefits of programmatically resetting the origin using the API. Thanks for the quick and thorough answers. Much appreciated,

Reply


Terms & Conditions