Issue: You encounter error HTTP_42 - Failing stage. Connection timeout as per configuration in HTTP Client destination.
Analysis: This is related to Maximum Request Time stage setting. Maximum Request Time can be viewed as allocated for the entire batch. While the stage is processing responses sequentially, if at any point the Maximum Request Time is exceeded, then that record and any records after it are hit with an HTTP_42. Please increase Maximum Request Time to a higher value to clear out this error.
However, from SDC 5.6.0 and below, we discover that there's a bug which makes Maximum Request Time setting conflicting with One request per Batch in HTTP Client Destination. This issue makes Maximum Request Time configuration not picked up when One request per Batch option is enabled.
Solution: This bug will be fixed in SDC 5.7.0 and above.
Workaround:
- 1st workaround: Please disable One request per Batch option.
- 2nd workaround (only applicable to SDC 5.4.0): Please replace these attached drivers in your SDC 5.4.0 install:
- streamsets-libs/streamsets-datacollector-basic-lib/lib/streamsets-datacollector-httpcommonlib-5.4.0.jar
- streamsets-libs/streamsets-datacollector-basic-lib/lib/streamsets-datacollector-basic-lib-5.4.0.jar
We currently don’t have the drivers handy for the rest of SDC versions. If you encounter timeout error that requires an increase in Maximum Request Time and you don’t want to disable One request per Batch, please raise a ticket with Support to provide suitable drivers for specific SDC version.