Is there a way to increase the amount of data that a HTTP client can return in StreamSets? It seems to be limited (I have maxed out the Max Object Length field and still get this error). The call returns the data OK in postman - the source file is only a 1.9Mb Excel with 29k rows.
WE ARE ON DATA COLLECTOR VERSION 5.0.0.
Best answer by Srinivasan Sankar
View originalThanks
I've set 'parser.limit=5335040' (5MB, the default is 1MB) on 'sdc.properties' SDC configuration file & restarted SDC service and the issue is resolved.
Credit goes to StreamSets Support team :)
Hi
The Max Object Length is limited by Integer.MAX_VALUE = 2^31 -1 = 2.147.483.647
So, if your Excel has more than 2.147.483.647, yes it might fail due to this limitation.
Something you might be able to do is read it as Text and use a Data Parser Processor to convert it to CSV. (I’ve not tried myself, but, it should be able to work)
Hope it helps
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
