Question

HTTP Client pagination calculation or configuration issue


I am attempting to call an API and use pagination in the HTTP Client origin.  I have everything configured and retrieving the first page of data, but the design of the API I’m using is odd.  I see the pagination options for Offset and Page, but the API I’m using actually uses those terms opposite of what Streamsets (and apparently the rest of the world) use.

The API call should be like so…

/api/myEndpoint?startDate=X&endDate=Y&limit=50&offset=0

So this would fetch 50 records at a time, and an offset value of 50 would USUALLY mean that I would be fetching records 51-100.  However the API I’m using would use an offset value of 1 to do the same (really we would call that page, not offset, in the rest of the world).  So - does this mean I can’t use the page or offset options in HTTP Client paging?

Alternatively I could do some gyrations with the Link in Response Field, but I don’t actually have one.  I could construct one with an expression or something, but the parameter in the Link to Response Field option is “Next page link field”, which I don’t have… I do have the totalRecords returned, so I could do some calculating but not sure where/how.

Can anyone offer any alternatives?  Thanks so much!


4 replies

Userlevel 3
Badge

Our HTTP Client origin docs have some examples of page vs. offset number that might be a good resource. Check those out and let us know if they help.

Yes @dima I’ve reviewed those, but my problem is that the API I’m using has different nomenclature - it uses “offset” and “page” in the opposite way than Streamsets (and apparently the rest of the world) so I think that’s causing my issue.  That’s what I’m trying to work around.  Streamsets and the rest of the planet use “offset” to mean 1-25, 26-50, 51-75, etc. when the limit is 25.  However the API I’m using unfortunately uses “1” for records 1-25, “2” for 26-50, etc.  Does that make sense?  That’s what I’m trying to work around.

Found my solution... Azure Data Factory.

Userlevel 3
Badge

Ah, thanks for letting us know. I think I came up with a solution here, but I guess it was a day too late.

All the best,

Reply