Hi there,
Any design suggestions to build a loop in StreamSets Data Collector pipelines ?
Example:
I have two API endpoints, the first one takes in a request to perform an activity and sends me an ID for the request placed.
As a second step, I need to poll a different endpoint with the ID obtained from the previous step to check the status of my earlier request (i.e. placed in the first step).
Ideally, I would like to loop that second endpoint in my StreamSets pipeline until I get the status as “succeeded” or “error”.
Any suggestions on how I achieve this in a data collector pipeline?
Cheers,
Srini