Question

HTTP processor - GET request pagination by page number always pulls the fist page only

  • 28 February 2023
  • 5 replies
  • 58 views
chandrautake
Rank

 We are working on a pipeline by using the http client processor to make HTTP GET request and the api  response is paginated but when using the pageNumber=${startAt} option its always pulling the first page

5 replies

saleempothiwala
Rank
Userlevel 4
Badge

Hi @chandrautake 

Make sure you have set the Multiple Values Behaviour to write all values to a list or to write to separate records.

 

When the property is set to default (write the first value only), the processor reads the first value in the first page of results only. The processor does not access the remaining pages of results, regardless of how pagination is configured.

 

 

chandrautake
Rank

@saleempothiwala I had this option enabled but still its fetching only the fist page. If you see the blow total record count is 1818 but its only one item in the list 

 

saleempothiwala
Rank
Userlevel 4
Badge

 

Please can you share your URL (remove any sensitive info)

 

I just want to check if you are using startAt value correctly. 

 

 

chandrautake
Rank
 
{

"name" : "conf.resourceUrl",

"value" : "${record:value('/base_url')}&pageNumber=${startAt}"

}
{

"name" : "conf.pagination.mode",

"value" : "BY_PAGE"

},  , {

"name" : "conf.pagination.startAt",

"value" : 1

}, {

"name" : "conf.pagination.resultFieldPath",

"value" : "/collection"

}, {

"name" : "conf.pagination.keepAllFields",

"value" : true

}, {

"name" : "conf.pagination.rateLimit",

"value" : 1000

},

{

"name" : "conf.missingValuesBehavior",

"value" : "PASS_RECORD_ON"

}, {

"name" : "conf.multipleValuesBehavior",

"value" : "ALL_AS_LIST"

}

 

saleempothiwala
Rank
Userlevel 4
Badge

@chandrautake the configurations looks good.

 

Please can you try using Dev Raw Data Source as origin (with option to stop after first batch) and provide URL hard coded with pageNumber = ${startAt} option added and see if it traverse through all 4 pages. 

Reply


Terms & Conditions