Skip to main content

HTTP Client stage and timeout handling


Victor
StreamSets Employee

This article aims to explain how to handle timeout errors when using HTTP Client under different scenarios. Given the nature of the HTTP timeouts, we cannot define one single configuration for all the timeout errors. Mainly we have two configuration parameters to set the timeout parameters: Request timeout and Read timeout. Once the timeout error occurs we must handle it in two different ways depending if it comes from an HTTP Status or a non-HTTP Status.

 

Timeout parameters

 

Request timeout

Configured from: HTTP Client > HTTP tab > Request timeout

This is the timeout we set for the HTTP Client to establish the connection with the server (until getting the first byte of a response from the target server).

Read timeout

Configured from: HTTP Client > HTTP tab > Read timeout

This is the timeout we set for the HTTP Client to retrieve the data we requested from the server (getting up to the last byte of a response from the target server).

 

Timeout Error Handling:

 

Per-Status Actions:

Configured from: HTTP Client > HTTP tab > Per-Status Actions

It captures HTTP statuses. Among many other statuses (like errors 404, 403…) it can also capture specific HTTP statuses related to timeouts such as error 408. Only timeouts with an HTTP status are captured using Per-Status Actions.

Timeouts:

Configured from: HTTP Client > Timeout tab

No timeout-like HTTP Status are treated as a timeout in the HTTP Client stages. Thus, actions per timeout are only responsible for taking care of network timeouts. Transport timeouts in the HTTP protocol reported as specific HTTP statuses need to be configured as per status actions to process them as if they were network timeouts.

 

The HTTP Protocol specifies processing a response in two minimum stages:

 

  1. Getting response: getting the first byte of a response from the target server

  2. Reading response: getting up to the last byte of a response from the target server


 

The types of network timeouts that we do control in HTTP Client (this should cover all kinds of timeouts you can get besides the ones reported as HTTP-specific statuses) are:

 

  • Connection: When receiving a connect timeout reaching the target server

  • Request: When a request to the target server cannot be completed due to a timeout

  • Response: When receiving a timeout while receiving a response from the target server

  • Read: When receiving a time-out while reading a response from the target server

  • Unknown: Exception marked as timeout but with no reason specified

 

Conclusion:

 

It is essential to understand that different scenarios may require different timeout configurations, and we cannot define a single configuration for all timeout errors. Thus, we will have to configure the timeout error behavior accordingly to the type of timeout we are facing: HTTP status or non-HTTP status timeout. By properly configuring the timeout parameters and understanding the different types of network timeouts, we can handle timeout errors effectively in the HTTP Client stage.

0 replies

Be the first to reply!

Reply