Looping in Data Collector pipelines

  • 24 June 2022
  • 1 reply

Userlevel 3
Badge +1

Hi there,


Any design suggestions to build a loop in StreamSets Data Collector pipelines ?




I have two API endpoints, the first one takes in a request to perform an activity and sends me an ID for the request placed.


As a second step, I need to poll a different endpoint with the ID obtained from the previous step to check the status of my earlier request (i.e. placed in the first step).


Ideally, I would like to loop that second endpoint in my StreamSets pipeline until I get the status as “succeeded” or “error”.


Any suggestions on how I achieve this in a data collector pipeline?




1 reply

Userlevel 2
Badge +1

@Srinivasan Sankar 

The custom function will be best solution for your case.

You can pass first API details to the function and then returned value can be pass to another function and return the result from that to streamsets.