Skip to main content

Hi!

We have a pipeline where we need to use two http stages. First we need to get all objects of type A and then for each A object we need to get all objects of type B.

1. First we have an http origin where we get all objects of type A. The only thing of interest in the response is the id of object A
2. For each A object, we make another http request (using a http client stage) to retrieve all the objects of type B. The url looks like this:
https://www.example.com/rest/1.0/foo/${record:value('/id')}/bar
i.e. we use the id of A which we got in the first stage.
   
We are not really interested in the records we get in stage one. The only thing we are interested in is the id which will be used in the URL in stage two.How can we discard all records received in stage one and just pass the id to stage two? Is it possible to write to an environment variable?

Any help would be appreciated!

Could you use a Field Remover, selecting to keep only the ‘/id’ field? 

Use the “Keep listed field” in the configuration of the Field Remover stage: https://docs.streamsets.com/portal/platform-datacollector/latest/datacollector/UserGuide/Processors/FieldRemover.html#task_c1j_btr_wq

Thank you, but that would mean that I still forward records from the first stage to the second. Ideally I don’t want that, I want to create all the records of interest in stage 2.

@lizlarsson 

 

 

In this scenario, it's advisable to insert a Field Remover processor between the source HTTP client and the HTTP client processor. Once you've filtered out the specific ID you're searching for using the Field Remover processor, you can then utilize this ID in the subsequent processor.

 

Kindly elaborate your issue in details ,so i can help you on it.

 

Thanks & Regards

Bikram_

Reply


Terms & Conditions