Skip to main content

I have a pipeline that is basically a pass-through REST service. It takes the input, routes based on URL with a stream splitter, sends out a REST call to another service of ours, gets the response and returns it to the sender. I am getting the output as expected.

However, I noticed that a call to the data collector REST endpoint results in 100+ http calls going out to our service (Call Deals / Generate Coupons in the image below).

When reviewing the snapshot, only 1 record shows. I can see from the service’s logs that many calls are coming in and giving a 200 response.

I am seeing the same behavior for both splits in the stream, so this is somehow related to the pipeline’s setup. Any ideas on what I could check?

 

 

Thanks!

Hi @xarnthehero, Is it possible to share a screenshot of the snapshot and also the exported pipeline json file?

Totally unrelated to your inquiry @xarnthehero. This is an awesome pipeline. Please share your pipeline with your first paragraph description in our “Show us your Pipeline contest”. You may submit as many as you wish.

Hi @Pradeep, thanks for the response!

Screenshot of the snapshot with the http call highlighted. Only 1 record is coming through, which is expected.

 

Snapshot json is attached. This instance called my endpoint 66 times but I don’t see those references in the json, just the 1 http call. The access log for the consuming service is attached as well showing it was called 66 times within two seconds.

 

If you can think of any other information that would help or anything to try, let me know!

For anyone stumbling upon this in the future, I found the issue. It has to do with HTTPClient “Multiple Values Behavior” setting being set to “All values as a list”. This field looks like it is related to pagination and pagination was set to None for my pipeline but for whatever reason this is resulting in many calls.

The HTTPClient documentation is well done and helpful - https://docs.streamsets.com/portal/datacollector/3.17.x/help//datacollector/UserGuide/Processors/HTTPClient.html

Reply


Terms & Conditions