I am attempting to call an API and use pagination in the HTTP Client origin.  I have everything configured and retrieving the first page of data, but the design of the API I’m using is odd.  I see the pagination options for Offset and Page, but the API I’m using actually uses those terms opposite of what Streamsets (and apparently the rest of the world) use.

The API call should be like so…

/api/myEndpoint?startDate=X&endDate=Y&limit=50&offset=0

So this would fetch 50 records at a time, and an offset value of 50 would USUALLY mean that I would be fetching records 51-100.  However the API I’m using would use an offset value of 1 to do the same (really we would call that page, not offset, in the rest of the world).  So - does this mean I can’t use the page or offset options in HTTP Client paging?

Alternatively I could do some gyrations with the Link in Response Field, but I don’t actually have one.  I could construct one with an expression or something, but the parameter in the Link to Response Field option is “Next page link field”, which I don’t have… I do have the totalRecords returned, so I could do some calculating but not sure where/how.

Can anyone offer any alternatives?  Thanks so much!