Skip to main content

I am essentially using a 2 step authentication to run a API call, first step is logging in and storing the cookie as a session id (post request), second is making a get request with that cookie to retrieve all the information. Now when I run this pipeline once it works fine. Problem is when I incorporate pagination, I need to ingest 1300 records and max records per api call is 500, so I will need to run in 3 batches.

I keep getting this error:

java.lang.ClassCastException: java.util.LinkedHashMap cannot be cast to java.util.List

 

when i set to pagination, is there anyway I can workaround this.

Note: Also I notice in data format for Http Client Processor, I only have option to click Json, but no further control e.g Array of json or multiple json objects, but when I use Http Client Origin, I have the option, is this a limitation within stremasets?

@saurabhsakpalbhp please share the config that you have used for pagination

@saurabhsakpalbhp please share the config that you have used for pagination

Sorry for late reply.

 

There is a limitation within streamsets that doesnt allow pagination for Http Clients that are processors, this is a bug that needs to be fixed. Http clients that are ‘Origin’ or ‘Destination’ have no issue. But ‘Processor’ you cant choose multiple Json objects, or Json Arrays after clicking on output of Json, hence giving me casting error for list shown in my first post.

 

Attached is the photos of configuration I am referring to.

Origin- You can choose Json→ Multiple Json (Allowing Pagination)

Processor- You can only choose Json, but no further option to choose array/list of Json, which is needed if you want to paginate. 

Reply


Terms & Conditions