StreamSets Support Team would like to inform you that the HTTP Client processor performs additional checks against the specified Batch Wait Time property starting with StreamSets Data Collector version 4.0.0.
Products Affected: StreamSets Data Collector
Releases Affected: StreamSets Data Collector 4.0.0
Users Affected: StreamSets Data Collector 4.0.0 pipelines that include the HTTP Client processor
Severity: Medium/Low
Description:
Starting with StreamSets Data Collector version 4.0.0, the HTTP Client processor performs additional checks against the specified Batch Wait Time property. In certain cases, this change can generate errors:
"Not able to finish all retries because the batch was timed out. Please increase the 'Batch Wait Time' or decrease the 'Base Backoff Interval'"
After upgrading from version 3.x to version 4.0.0 or later, verify that pipelines that include the HTTP Client processor perform as expected.
The Batch Wait Time property defines the maximum amount of time that the processor uses to process all HTTP requests for a single record. When the processing for a record exceeds the specified batch wait time, the output records are passed to the stage for error handling.
In previous releases, the HTTP Client processor only checked the batch wait time before each HTTP request. As a result, the processor did not always notice when the processing time exceeded the batch wait time.
Starting with version 4.0.0, the HTTP Client processor checks the batch wait time before and after every request. As a result, the processor may generate more errors than in previous releases.
Also, in previous releases, the default value for Batch Wait Time was 2,000 milliseconds. Starting with version 4.0.0, the default value is 100,000 milliseconds. When you upgrade from version 3.x to version 4.0.0 or later, the Batch Wait Time property in the HTTP Client processor is set to the new default of 100,000 milliseconds, unless you changed the property from the default.
Please find the details also in Review HTTP Client Processor Pipelines in our documentation.
Immediate action required: Yes
Fix: Not an issue
Resolution: After upgrading StreamSets Data Collector from version 3.x to version 4.0.0 or later, verify that pipelines that include the HTTP Client processor perform as expected. If you want the processor to wait for all HTTP requests to complete, increase the Batch Wait Time as needed.
For your questions, please Submit a request via our Zendesk support portal or contact us at support@streamsets.com.
