Skip to main content
Solved

HTTP Server - Max Request Size

  • January 13, 2023
  • 3 replies
  • 101 views
T
Fan

hi, i am using in one pipeline as origin HTTP Server with value Max Request Size = 100MB. 

I start to getting errors like this: 
CONTAINER_0001 - com.streamsets.pipeline.api.ext.io.OverrunException: Reader exceeded the read limit '100000000'

Is there anything from configuration perspective how to fix it, to allow bigger records?

Thanks

 

 

 

 

Best answer by Dimas Cabré i Chacón

To configure larger values you need to set property

 

com.streamsets.httpserverpushsource.maxrequest.mb
 

in your SDC startup script, like 

-Dcom.streamsets.httpserverpushsource.maxrequest.mb=200
View original
Did this topic help you find an answer to your question?

3 replies

Dimas Cabré i Chacón
StreamSets Employee
Forum|alt.badge.img

To configure larger values you need to set property

 

com.streamsets.httpserverpushsource.maxrequest.mb
 

in your SDC startup script, like 

-Dcom.streamsets.httpserverpushsource.maxrequest.mb=200
T
Fan
  • thirty2Fan
  • Author
  • Fan
  • 2 replies
  • January 15, 2023
Dimas Cabré i Chacón wrote:

To configure larger values you need to set property

 

com.streamsets.httpserverpushsource.maxrequest.mb
 

in your SDC startup script, like 

-Dcom.streamsets.httpserverpushsource.maxrequest.mb=200

Thanks, it is working. 
Just to be sure, is it a size per record?

Dimas Cabré i Chacón
StreamSets Employee
Forum|alt.badge.img

@thirty2 Glad to hear it worked. Size is per request, which normally is one request per record.

Reply


Terms & Conditions