Question

Keep all fields creating duplicate records in http client origin


I have an http client origin which gives a json response. Pipeline uses pagination (by page number). When I am enabling ‘Keep all fields’ option in http client it is creates the duplicates of first record in every page. Say I have 10 records in the file, it writes first record 10 times in the same, 1st record in the second page 10 times and so on. basically it repeats first record for the entire page. Any way to fix this issue? We need to ‘Keep all fields’ enabled to get page properties for processing within the job. 


2 replies

Userlevel 3
Badge

Hello @AJ20 Could you please share the sample json response output. How does it look like? 

Userlevel 2
Badge +1

@AJ20 

can you please try to select the option  as “All values as a list” in HTTP origin , i believe the current select is     “Split into multiple records” in HTTP origin , that’s the reason you are getting duplicate entries in pagination.

Reply