Skip to main content
Solved

One REST request gets turned into multiple outbound requests

  • February 15, 2022
  • 4 replies
  • 160 views
xarnthehero
Fan

I have a pipeline that is basically a pass-through REST service. It takes the input, routes based on URL with a stream splitter, sends out a REST call to another service of ours, gets the response and returns it to the sender. I am getting the output as expected.

However, I noticed that a call to the data collector REST endpoint results in 100+ http calls going out to our service (Call Deals / Generate Coupons in the image below).

When reviewing the snapshot, only 1 record shows. I can see from the service’s logs that many calls are coming in and giving a 200 response.

I am seeing the same behavior for both splits in the stream, so this is somehow related to the pipeline’s setup. Any ideas on what I could check?

 

 

Thanks!

Best answer by xarnthehero

For anyone stumbling upon this in the future, I found the issue. It has to do with HTTPClient “Multiple Values Behavior” setting being set to “All values as a list”. This field looks like it is related to pagination and pagination was set to None for my pipeline but for whatever reason this is resulting in many calls.

The HTTPClient documentation is well done and helpful - https://docs.streamsets.com/portal/datacollector/3.17.x/help//datacollector/UserGuide/Processors/HTTPClient.html

View original
Did this topic help you find an answer to your question?

4 replies

Pradeep
StreamSets Employee
Forum|alt.badge.img+1
  • PradeepStreamSets Employee
  • StreamSets Employee
  • 48 replies
  • March 9, 2022

Hi @xarnthehero, Is it possible to share a screenshot of the snapshot and also the exported pipeline json file?

Drew Kreiger
1 person likes this
  • Drew Kreiger
    Drew Kreiger
Drew Kreiger
Rock star
Forum|alt.badge.img
  • Drew KreigerRock star
  • Senior Community Builder at StreamSets
  • 95 replies
  • March 9, 2022

Totally unrelated to your inquiry @xarnthehero. This is an awesome pipeline. Please share your pipeline with your first paragraph description in our “Show us your Pipeline contest”. You may submit as many as you wish.

xarnthehero
Fan
  • xarntheheroFan
  • Author
  • Fan
  • 3 replies
  • March 11, 2022

Hi @Pradeep, thanks for the response!

Screenshot of the snapshot with the http call highlighted. Only 1 record is coming through, which is expected.

 

Snapshot json is attached. This instance called my endpoint 66 times but I don’t see those references in the json, just the 1 http call. The access log for the consuming service is attached as well showing it was called 66 times within two seconds.

 

If you can think of any other information that would help or anything to try, let me know!

2 Attachments
Drew Kreiger
1 person likes this
  • Drew Kreiger
    Drew Kreiger
xarnthehero
Fan
  • xarntheheroFan
  • Author
  • Fan
  • 3 replies
  • Answer
  • April 13, 2022

For anyone stumbling upon this in the future, I found the issue. It has to do with HTTPClient “Multiple Values Behavior” setting being set to “All values as a list”. This field looks like it is related to pagination and pagination was set to None for my pipeline but for whatever reason this is resulting in many calls.

The HTTPClient documentation is well done and helpful - https://docs.streamsets.com/portal/datacollector/3.17.x/help//datacollector/UserGuide/Processors/HTTPClient.html

Reply


Terms & Conditions