I have an http client origin which gives a json response. Pipeline uses pagination (by page number). When I am enabling ‘Keep all fields’ option in http client it is creates the duplicates of first record in every page. Say I have 10 records in the file, it writes first record 10 times in the same, 1st record in the second page 10 times and so on. basically it repeats first record for the entire page. Any way to fix this issue? We need to ‘Keep all fields’ enabled to get page properties for processing within the job.
Already have an account? Login
Login to the community
No account yet? Create an account
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.