Solved

HTTP Client Origin - Max Object Length Exceeded

  • 17 May 2022
  • 2 replies
  • 63 views

Userlevel 3
Badge +1

Is there a way to increase the amount of data that a HTTP client can return in StreamSets? It seems to be limited (I have maxed out the Max Object Length field and still get this error). The call returns the data OK in postman - the source file is only a 1.9Mb Excel with 29k rows.

WE ARE ON DATA COLLECTOR VERSION 5.0.0.

icon

Best answer by Srinivasan Sankar 18 May 2022, 11:26

View original

2 replies

Userlevel 3
Badge +1

Thanks @alex.sanchez,

I've set 'parser.limit=5335040' (5MB, the default is 1MB) on 'sdc.properties' SDC configuration file & restarted SDC service and the issue is resolved.

Credit goes to StreamSets Support team :)

Userlevel 2
Badge

Hi @Srinivasan Sankar,

The Max Object Length is limited by Integer.MAX_VALUE = 2^31 -1 = 2.147.483.647

So, if your Excel has more than 2.147.483.647, yes it might fail due to this limitation.

Something you might be able to do is read it as Text and use a Data Parser Processor to convert it to CSV. (I’ve not tried myself, but, it should be able to work)

Hope it helps

Reply