I have an http client origin which gives a json response. Pipeline uses pagination (by page number). When I am enabling ‘Keep all fields’ option in http client it is creates the duplicates of first record in every page. Say I have 10 records in the file, it writes first record 10 times in the same, 1st record in the second page 10 times and so on. basically it repeats first record for the entire page. Any way to fix this issue? We need to ‘Keep all fields’ enabled to get page properties for processing within the job.
Already have an account? Login
Login to the community
No account yet? Create an account
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
can you please try to select the option as “All values as a list” in HTTP origin , i believe the current select is “Split into multiple records” in HTTP origin , that’s the reason you are getting duplicate entries in pagination.
@AJ20 Could you please share the sample json response output. How does it look like?