I see the option in the Streamsets canvas:
- Start pipeline
- Reset origin and start
- Start with parameters
Is it possible to combine the last 2 options?
- Reset origin and start with parameters
Best answer by Srinivasan SankarView original
If you are OK to achieve your requirement via programmatically, then it's trivial to use the SDC REST APIs and achieve the same. You need to call Rest Origin REST API followed by Start pipeline API with runtime variables in POST body.
The other option would be,
Below is an example where a Pipeline Finisher is used in a pipeline. The configuration does not reset the Origin but you could reset the Origin by enabling the check-box.
Note - the Pipeline Finisher stops the pipeline and resets the Origin when there are no more data to process from the source. All other events are discarded.
Both of the above answers are excellent. I chose the second as “best” because right now I am working in the UI. But I definitely see the benefits of programmatically resetting the origin using the API. Thanks for the quick and thorough answers. Much appreciated,