HTTP client step- pagination not working for SAOP API

  • 8 August 2022
  • 2 replies

I am trying to call a SOAP API which requires page number and number of records to be mentioned in request body.

We have multiple pages in response however streamsets pipeline just ingest the first page.


resource URL - just URL of the API call , no variable for pagination here .

Headers : blank

mode : batch

HTTP Method : POST

Request body : complete request body with page number 


<Response Filter>

</Response Filter>



By page number 

initial page :1 

rest default 


can someone help how to retrieve all available pages.

tried using value ${startAt} for page in request body.



2 replies

Was this a 2 step authentication?

By the way i think you should be using a get request not a post.