We are working on a pipeline by using the http client processor to make HTTP GET request and the api response is paginated but when using the pageNumber=${startAt}
option its always pulling the first page
Hi
Make sure you have set the Multiple Values Behaviour to write all values to a list or to write to separate records.
When the property is set to default (write the first value only), the processor reads the first value in the first page of results only. The processor does not access the remaining pages of results, regardless of how pagination is configured.
Please can you share your URL (remove any sensitive info)
I just want to check if you are using startAt value correctly.
{
"name" : "conf.resourceUrl",
"value" : "${record:value('/base_url')}&pageNumber=${startAt}"
}
{
"name" : "conf.pagination.mode",
"value" : "BY_PAGE"
}, , {
"name" : "conf.pagination.startAt",
"value" : 1
}, {
"name" : "conf.pagination.resultFieldPath",
"value" : "/collection"
}, {
"name" : "conf.pagination.keepAllFields",
"value" : true
}, {
"name" : "conf.pagination.rateLimit",
"value" : 1000
},
{
"name" : "conf.missingValuesBehavior",
"value" : "PASS_RECORD_ON"
}, {
"name" : "conf.multipleValuesBehavior",
"value" : "ALL_AS_LIST"
}
Please can you try using Dev Raw Data Source as origin (with option to stop after first batch) and provide URL hard coded with pageNumber = ${startAt} option added and see if it traverse through all 4 pages.
Reply
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.