Skip to main content

Hi there,

 

Any design suggestions to build a loop in StreamSets Data Collector pipelines ?

 

 

Example:

I have two API endpoints, the first one takes in a request to perform an activity and sends me an ID for the request placed.

 

As a second step, I need to poll a different endpoint with the ID obtained from the previous step to check the status of my earlier request (i.e. placed in the first step).

 

Ideally, I would like to loop that second endpoint in my StreamSets pipeline until I get the status as “succeeded” or “error”.

 

Any suggestions on how I achieve this in a data collector pipeline?

 

Cheers,

Srini

@Srinivasan Sankar 

The custom function will be best solution for your case.

You can pass first API details to the function and then returned value can be pass to another function and return the result from that to streamsets.

 


Hi, Actually making shell executor as as processor will help to perform this activities and connect to different stages again based on the return codes from shell execution

 

@Srinivasan Sankar 

@Bikram


Reply