Skip to main content

Is there a way to increase the amount of data that a HTTP client can return in StreamSets? It seems to be limited (I have maxed out the Max Object Length field and still get this error). The call returns the data OK in postman - the source file is only a 1.9Mb Excel with 29k rows.

WE ARE ON DATA COLLECTOR VERSION 5.0.0.

Hi @Srinivasan Sankar,

The Max Object Length is limited by Integer.MAX_VALUE = 2^31 -1 = 2.147.483.647

So, if your Excel has more than 2.147.483.647, yes it might fail due to this limitation.

Something you might be able to do is read it as Text and use a Data Parser Processor to convert it to CSV. (I’ve not tried myself, but, it should be able to work)

Hope it helps

Thanks @alex.sanchez,

I've set 'parser.limit=5335040' (5MB, the default is 1MB) on 'sdc.properties' SDC configuration file & restarted SDC service and the issue is resolved.

Credit goes to StreamSets Support team :)

Reply


Terms & Conditions