Question

Problem with Pagination

  • 22 September 2023
  • 1 reply
  • 30 views
southwr62
Rank

Hi, 

I am using the HTTP client origin, and since I need to load a lot of records, I need to use pagination.my url ends with ‘auditLogs?offset=${startAt}&pagesize=5&isascorder=false&order=id’ cannot give more for confidentiallity reasons. This is a post request

when I set pagination to offset or page number, all I get is the first 5 records, offset is actually used to denote the start record of the data returned, not a page in this instance.

If I set pagesize to 1, which I think is how I need to do this, I only get 1 result it doesn’t seem to continue, I have tried with both page and offset set as the pagination method.

any ideas on what is wrong are welcomed

1 reply

M
Rank

Hi,

 

The HTTP Client origin provides a ${startAt} variable that you can use in the URL as the value of the page or offset. For example, your resource URL might be any of the following:

  • http://webservice/object?limit=15&offset=${startAt}
  • https://myapp.com/product?limit=5&since=${startAt}
  • https://myotherapp.com/api/v1/products?page=${startAt}

When the pipeline starts, the HTTP Client stage uses the value of the Initial Page/Offset property as the ${startAt} variable value. After the stage reads a page of results, the stage increments the ${startAt} variable by one if using page number pagination or by the number of records read from the page if using offset number pagination.

 

Please refer this below Streamsets docs url and check once all assigned correctly and give a another try.

https://docs.streamsets.com/platform-datacollector/latest/datacollector/UserGuide/Origins/HTTPClient.html#concept_wk4_bjz_5r

Reply


Terms & Conditions