Looping in Data Collector pipelines

Hi there,

Any design suggestions to build a loop in StreamSets Data Collector pipelines ?

Example:

I have two API endpoints, the first one takes in a request to perform an activity and sends me an ID for the request placed.

As a second step, I need to poll a different endpoint with the ID obtained from the previous step to check the status of my earlier request (i.e. placed in the first step).

Ideally, I would like to loop that second endpoint in my StreamSets pipeline until I get the status as “succeeded” or “error”.

Any suggestions on how I achieve this in a data collector pipeline?

Cheers,

Srini