Solved

HTTP Server - Max Request Size

  • 13 January 2023
  • 3 replies
  • 69 views

hi, i am using in one pipeline as origin HTTP Server with value Max Request Size = 100MB. 

I start to getting errors like this: 
CONTAINER_0001 - com.streamsets.pipeline.api.ext.io.OverrunException: Reader exceeded the read limit '100000000'

Is there anything from configuration perspective how to fix it, to allow bigger records?

Thanks

 

 

 

 

icon

Best answer by Dimas Cabré i Chacón 13 January 2023, 12:32

View original

3 replies

Userlevel 2
Badge

@thirty2 Glad to hear it worked. Size is per request, which normally is one request per record.

To configure larger values you need to set property

 

com.streamsets.httpserverpushsource.maxrequest.mb
 

in your SDC startup script, like 

-Dcom.streamsets.httpserverpushsource.maxrequest.mb=200

Thanks, it is working. 
Just to be sure, is it a size per record?

Userlevel 2
Badge

To configure larger values you need to set property

 

com.streamsets.httpserverpushsource.maxrequest.mb
 

in your SDC startup script, like 

-Dcom.streamsets.httpserverpushsource.maxrequest.mb=200

Reply