Question

Looping in Data Collector pipelines

  • 24 June 2022
  • 2 replies
  • 374 views

Userlevel 3
Badge +1

Hi there,

 

Any design suggestions to build a loop in StreamSets Data Collector pipelines ?

 

 

Example:

I have two API endpoints, the first one takes in a request to perform an activity and sends me an ID for the request placed.

 

As a second step, I need to poll a different endpoint with the ID obtained from the previous step to check the status of my earlier request (i.e. placed in the first step).

 

Ideally, I would like to loop that second endpoint in my StreamSets pipeline until I get the status as “succeeded” or “error”.

 

Any suggestions on how I achieve this in a data collector pipeline?

 

Cheers,

Srini


2 replies

Userlevel 5
Badge +1

@Srinivasan Sankar 

The custom function will be best solution for your case.

You can pass first API details to the function and then returned value can be pass to another function and return the result from that to streamsets.

 

Hi, Actually making shell executor as as processor will help to perform this activities and connect to different stages again based on the return codes from shell execution

 

@Srinivasan Sankar 

@Bikram

Reply