Skip to main content

We have a multi-thread runner listening for new data on an S3 bucket and processing and filtering the data. Our pipeline seems to work well for a few hours, but gets this error below. We are getting about 5-10MB of data per second and that gets filtered down to about 2% of data through our processing and filtering. Is this due to too much data or a configuration problem?

 

Pipeline Status: RUN_ERROR: S3_SPOOLDIR_26 - S3 runner failed. Reason org.apache.http.NoHttpResponseException: The target server failed to respond

Anyone have a resolution to this issue? 


Reply