From 394f4a02a5eda0e7df7f25d80fb7729aabde5a97 Mon Sep 17 00:00:00 2001 From: Osura Viduranga Date: Tue, 30 Jun 2026 13:54:08 +0530 Subject: [PATCH] Update gateway timeout docs with new configs --- .../api-gateway/setup/upstream-timeouts.md | 310 ++++++++++++++---- en/docs/llms.txt | 2 +- en/mkdocs.yml | 2 +- 3 files changed, 251 insertions(+), 63 deletions(-) diff --git a/en/docs/api-gateway/setup/upstream-timeouts.md b/en/docs/api-gateway/setup/upstream-timeouts.md index e0e9820a5..8fa3326dc 100644 --- a/en/docs/api-gateway/setup/upstream-timeouts.md +++ b/en/docs/api-gateway/setup/upstream-timeouts.md @@ -1,83 +1,92 @@ --- -title: "Configure Upstream Timeouts" -description: "Configure the upstream TCP connection timeout in API Platform Gateway to protect against slow or unreachable backend services." +title: "Configure Gateway Timeouts" +description: "Configure router-level and API-level timeouts (connect, route, idle, and HTTP connection manager) in the API Platform Gateway to protect against slow or unreachable backends and slow clients." canonical_url: https://wso2.com/api-platform/docs/api-gateway/setup/upstream-timeouts/ md_url: https://wso2.com/api-platform/docs/api-gateway/setup/upstream-timeouts.md tags: - api-gateway - configuration - networking + - resiliency author: WSO2 API Platform Documentation Team -last_updated: 2026-06-17 +last_updated: 2026-06-30 content_type: "how-to" --- -# Gateway Upstream Timeouts +# Gateway Timeouts -This guide explains how to configure **upstream connect timeouts** for the API Platform Gateway router so that requests to slow or unreachable backends fail within a predictable time instead of hanging indefinitely. +This guide explains how to configure **timeouts** for the API Platform Gateway so that requests to slow or unreachable backends, and slow downstream clients, fail within a predictable time instead of hanging indefinitely. -## Overview +Timeouts are configured at **two levels**: -When the router forwards a request to a backend service, it must first establish a TCP connection to the upstream host. The **connect timeout** controls how long the router waits for this connection to be established. +| Level | Where it is configured | Scope | Typical use | +|-------|------------------------|-------|-------------| +| **Router level** | The gateway controller's `config.toml` (`[router.*]`) | Global defaults applied to **all** traffic on the gateway | Operator-wide baselines (connect timeout, default route timeout, downstream/HCM timeouts) | +| **API level** | The API definition — the `resilience` block (RestApi, LLM Provider, LLM Proxy) and `upstreamDefinitions[].timeout.connect` (RestApi) | A single API, its routes, and its upstream clusters | Per-API/per-route overrides for backends that are faster or slower than the global default | -- If the backend accepts connections within the configured time, the request proceeds as normal. -- If the connection cannot be established before the timeout elapses, the router fails the request with an upstream timeout error. +The router level establishes the **defaults**. The API level **overrides** those defaults for a specific API (and, for REST APIs, a specific operation). When an API does not specify a value, the router-level default applies. -This timeout acts as a **global default** for upstream connections and is a key part of the gateway’s resiliency story: it helps protect clients from slow or unreachable backends and frees up resources more quickly when backends are unhealthy. +## Router-level timeouts (config.toml) -## How the upstream connect timeout works +Router-level timeouts are global defaults that apply to every request handled by the gateway, regardless of which API serves it. They are configured in the gateway controller configuration. You can use `gateway/configs/config-template.toml` as a reference when creating your own `config.toml`. -At a high level: +There are two groups: -1. A client sends a request to the gateway. -2. The router selects the appropriate upstream cluster for the request. -3. The router attempts to establish a TCP connection to one of the upstream endpoints. -4. If the connection is not established within `connect_timeout_ms`, the attempt is aborted and the request fails with an upstream timeout. +- **Upstream timeouts** (`[router.upstream.timeouts]`) — govern the connection to, and the response from, the **backend**. +- **HTTP Connection Manager (HCM) timeouts** (`[router.http_listener.timeouts]`) — govern the **downstream** connection between the client and the gateway. -This is specifically about **connection establishment time**. It is separate from any overall request/route timeouts that may control how long the router waits for responses after a connection is established. +### Upstream timeouts -## Configuring the global connect timeout +```toml +[router.upstream.timeouts] +route_timeout_ms = 60000 # Default per-route request→response timeout +route_idle_timeout_ms = 300000 # Default per-route stream idle timeout +connect_timeout_ms = 5000 # TCP connection establishment timeout +``` -The global upstream connect timeout is configured in the gateway controller configuration under the `router.upstream.timeouts` block. +| Setting | Default | Maps to (Envoy) | Description | +|---------|---------|-----------------|---------| +| `connect_timeout_ms` | `5000` | Cluster `connect_timeout` | How long the router waits to **establish a TCP connection** to an upstream endpoint before failing the request. | +| `route_timeout_ms` | `60000` | `RouteAction.timeout` | Default maximum time for the **entire request→upstream-response** on a route. This is the default that an API's `resilience.timeout` overrides. | +| `route_idle_timeout_ms` | `300000` | `RouteAction.idle_timeout` | Default **per-route stream idle** timeout. This is the default that an API's `resilience.idleTimeout` overrides. | -You can use `gateway/configs/config-template.toml` as a reference when creating your own `config.toml`. +!!! note + All three values must be **positive** (greater than `0`). To disable a *route* timeout for a specific API, set its `resilience` value to `0s` (see [API-level timeouts](#api-level-timeouts)). -### Example configuration (TOML) +### HTTP Connection Manager (downstream) timeouts -The following example shows how to set the upstream connect timeout to 6 seconds (6000 ms): +These govern the connection between the **client and the gateway**. A value of `"0s"` disables the corresponding timeout. ```toml -[router] -gateway_host = "*" -listener_port = 8080 -https_enabled = true -https_port = 8443 - -[router.upstream.timeouts] -connect_timeout_ms = 6000 +# HTTP Connection Manager (downstream) timeouts +[router.http_listener.timeouts] +request_timeout = "0s" # Max duration for the entire downstream request +request_headers_timeout = "0s" # Max duration to receive the complete request headers +stream_idle_timeout = "5m" # Idle timeout for a single HTTP stream/request +idle_timeout = "1h" # Idle timeout for the downstream connection ``` -In this example: +| Setting | Default | Maps to (Envoy) | Description | +|---------|---------|-----------------|---------| +| `request_timeout` | `0s` (disabled) | HCM `request_timeout` | Max time for the gateway to receive the **entire** downstream request. | +| `request_headers_timeout` | `0s` (disabled) | HCM `request_headers_timeout` | Max time to receive the **complete request headers**. | +| `stream_idle_timeout` | `5m` | HCM `stream_idle_timeout` | Idle timeout for a single HTTP **stream/request**. | +| `idle_timeout` | `1h` | HCM `common_http_protocol_options.idle_timeout` | Idle timeout for the downstream **connection**. | -- `connect_timeout_ms = 6000` means the router will wait up to 6 seconds for a TCP connection to the upstream before failing the request. -- If you do not override this value, the default from the controller configuration is used (typically 5000 ms). +Durations use Go duration syntax with a single unit (for example `"30s"`, `"500ms"`, `"5m"`, `"1h"`). If a value is left unset, the Envoy default for that field applies. -### Changing the value in different deployments +!!! warning + `idle_timeout` can also be disabled by setting it to zero explicitly. Disabling it has a high likelihood of yielding connection leaks (for example, due to lost TCP FIN packets). -#### Standalone / local configuration +### Setting router-level timeouts in different deployments -For local or non-Kubernetes deployments, you configure the timeout directly in `gateway/configs/config.toml` using the same structure as shown above: +#### Standalone / local -```toml -[router.upstream.timeouts] -connect_timeout_ms = 5000 -``` +Configure the timeouts directly in `gateway/configs/config.toml` using the structure shown above. #### Kubernetes with Helm -When deploying the gateway via the Helm chart, the same setting is controlled through Helm values under the `gateway.config.router.upstream.timeouts` section. - -Example Helm `values.yaml` snippet: +When deploying the gateway via the Helm chart, the same settings are controlled through Helm values under `gateway.config.router.*`. Example `values.yaml` snippet: ```yaml gateway: @@ -86,40 +95,219 @@ gateway: upstream: timeouts: connect_timeout_ms: 5000 + route_timeout_ms: 60000 + route_idle_timeout_ms: 300000 + http_listener: + timeouts: + request_timeout: "0s" + request_headers_timeout: "0s" + stream_idle_timeout: "5m" + idle_timeout: "1h" +``` + +The chart renders these values into the generated `config.toml` used by the gateway controller. + +## API-level timeouts + +Two of the timeout layers are naturally per-API and can be tuned directly in the API definition, each through its own field: + +| Surface | Field(s) | Maps to (Envoy) | Overrides router default | +|---------|----------|-----------------|--------------------------| +| `upstreamDefinitions[].timeout.connect` | `connect` | Cluster `connect_timeout` | `connect_timeout_ms` | +| `resilience` block | `timeout`, `idleTimeout` | `RouteAction.timeout` / `idle_timeout` | `route_timeout_ms` / `route_idle_timeout_ms` | + +The HCM (downstream) timeouts have no per-API equivalent and remain router-level only. + +### Connect timeout per upstream (`upstreamDefinitions[].timeout.connect`) + +When an API routes through a named **upstream definition**, that definition can set its own `connect` timeout, which overrides the router-level `connect_timeout_ms` for the cluster built from it. Because the setting lives on the upstream definition, different upstreams used by the same API can have different connect timeouts. + +```yaml +apiVersion: gateway.api-platform.wso2.com/v1 +kind: RestApi +metadata: + name: orders-api +spec: + displayName: Orders API + version: v1.0 + context: /orders/$version + upstreamDefinitions: + - name: orders-backend + timeout: + connect: 6000ms # overrides connect_timeout_ms for this cluster + upstreams: + - url: http://orders.internal:8080 + upstream: + main: + ref: orders-backend # this API routes through the orders-backend cluster + operations: + - method: GET + path: /list ``` -The chart then renders this value into the generated `config.toml` used by the gateway controller. +- `connect` is a duration string (for example `6000ms`, `5s`). +- It governs only **TCP connection establishment** (plus the TLS handshake for HTTPS upstreams) — not the response wait, which is the route timeout below. +- An API that uses a direct `upstream.main.url` (instead of an `upstreamDefinitions` ref) uses the router-level `connect_timeout_ms`. + +### Route timeouts (via `resilience` block) + +The `resilience` block lets an individual API override the router-level **route** timeouts for its own traffic. It maps to Envoy's `RouteAction` timeouts and is available on `RestApi`, `LlmProvider`, and `LlmProxy` resources. + +It supports two fields: + +| Field | Maps to (Envoy) | Overrides router default | +|-------|-----------------|--------------------------| +| `timeout` | `RouteAction.timeout` | `route_timeout_ms` | +| `idleTimeout` | `RouteAction.idle_timeout` | `route_idle_timeout_ms` | + +!!! note "Duration format" + `timeout` and `idleTimeout` take single-unit duration strings (for example `30s`, `500ms`, `1.5m`). Use `0s` to disable a timeout, or omit the field to fall back to the router-level default. Compound (`1h30m`), negative (`-30s`), and unitless (`0`, `30`) values are rejected. + +### REST APIs: API level and operation level + +For `RestApi`, the `resilience` block can be set at **two levels**: + +- **API level** — applies to every operation/route of the API. +- **Operation level** — applies to that single operation and overrides the API-level value. + +Precedence is **most-specific-wins, per field**: an operation's `timeout` overrides the API's `timeout`, while still inheriting the API's `idleTimeout` if the operation does not set its own. If neither level sets a field, the router-level default applies. + +```yaml +apiVersion: gateway.api-platform.wso2.com/v1 +kind: RestApi +metadata: + name: orders-api +spec: + displayName: Orders API + version: v1.0 + context: /orders/$version + upstream: + main: + url: https://backend.example.com + # API-level: applies to all operations unless overridden. + resilience: + timeout: 15s + idleTimeout: 30s + operations: + - method: GET + path: /list + - method: POST + path: /reports + # Operation-level override: this slow report endpoint gets a longer + # route timeout, but still inherits the API-level idleTimeout (30s). + resilience: + timeout: 60s +``` + +Resolution for the example above: + +| Route | `timeout` | `idleTimeout` | +|-------|-----------|---------------| +| `GET /list` | `15s` (API level) | `30s` (API level) | +| `POST /reports` | `60s` (operation level) | `30s` (inherited from API level) | + +### LLM Provider and LLM Proxy: API level only + +For `LlmProvider` and `LlmProxy`, the `resilience` block is supported at the **API level only**. LLM routes are generated by the gateway from the access-control configuration and policy attachments (rather than authored as explicit operations), so there is no operation-level override. A single API-level block applies to **all** routes generated for the LLM API. + +```yaml +apiVersion: gateway.api-platform.wso2.com/v1 +kind: LlmProvider +metadata: + name: openai-provider +spec: + displayName: OpenAI Provider + version: v1.0 + template: openai + context: /openai + upstream: + url: https://api.openai.com/v1 + accessControl: + mode: deny_all + exceptions: + - path: /chat/completions + methods: [POST] + # Applies to every route generated for this provider. + resilience: + timeout: 60s + idleTimeout: 30s +``` + +```yaml +apiVersion: gateway.api-platform.wso2.com/v1 +kind: LlmProxy +metadata: + name: openai-proxy +spec: + displayName: OpenAI Proxy + version: v1.0 + context: /openai-proxy + provider: + id: openai-provider + resilience: + timeout: 75s +``` + +!!! note "LLM Proxy double hop" + A request to a proxy traverses two routes — `client → proxy route → (loopback) → provider route → backend`. The **proxy's** `resilience.timeout` bounds the whole proxied call, while the backing **provider's** `resilience.timeout` bounds the provider→backend call. For a meaningful end-to-end budget, keep the proxy timeout greater than or equal to the provider timeout; the shorter of the two effectively wins. ## Practical guidance -When tuning `connect_timeout_ms`, consider the characteristics of your backends and network: +### Connect timeout (`connect_timeout_ms`) -- **Decrease the timeout** when: - - Backends are expected to be highly available and respond quickly to connection attempts. - - You want to fail fast when targets are unhealthy or misconfigured. - - You want to reduce the time resources are tied up on connection attempts to non-responsive hosts. +- **Decrease** when backends are highly available and you want to fail fast on unhealthy or misconfigured targets, freeing resources quickly. +- **Increase** when backends sit behind slower networks/load balancers, or may experience cold starts or scaling events that briefly delay connection establishment. +- Avoid setting it too low — it may cause **false-positive timeouts** during short periods of backend slowness or network jitter. +- Set the **router default** for the common case, and override **per upstream** via `upstreamDefinitions[].timeout.connect` (see [Connect timeout per upstream](#connect-timeout-per-upstream-upstreamdefinitionstimeoutconnect)) when a specific backend needs a different connection budget. -- **Increase the timeout** when: - - Backends sit behind slower networks or load balancers that may take longer to accept connections. - - Backends may experience occasional cold starts or scaling events that briefly delay connection establishment. +### Route timeout (`route_timeout_ms` / `resilience.timeout`) -Be cautious about setting the timeout too low—it may cause **false-positive timeouts** during short periods of backend slowness or network jitter. +- Set the **router default** to a sane upper bound for typical backends, and use **API/operation-level** `resilience.timeout` for endpoints that are known to be faster (tighter budget) or legitimately slower (e.g. report generation, LLM completions). +- For streaming/long-lived responses (e.g. SSE from LLM backends), be mindful of `idleTimeout`: a long total `timeout` with a short `idleTimeout` can still cut a slow token stream. Leave `idleTimeout` generous (or unset) for streaming providers. -## Example scenario +### HCM/downstream timeouts -Consider an API whose upstream backend may take a few seconds to accept new connections during peak load. You want the gateway to: +- `request_headers_timeout` is a useful protection against slow-header (Slowloris-style) clients; enable it with a small value (for example `"5s"`) if your clients are expected to send headers promptly. +- `request_timeout` is disabled by default; enable it only if you want a hard cap on how long the gateway will spend receiving an entire downstream request. -- Allow a bit more time than the default for connections to succeed. -- Still fail requests in a bounded time when the backend is unreachable. +## Example scenarios -You can configure: +### A slow-to-connect backend + +A backend may take a few seconds to accept new connections during peak load. Give connections more time than the default while still failing in a bounded time when the backend is unreachable: ```toml [router.upstream.timeouts] connect_timeout_ms = 6000 ``` -With this configuration: +The router gives each upstream connection attempt up to 6 seconds; if the backend is down, requests fail after ~6 seconds (`503`) instead of hanging. + +### One slow endpoint on an otherwise fast API + +The API as a whole should respond within 5 seconds, but a single report endpoint legitimately takes longer: + +```yaml +spec: + resilience: + timeout: 5s + operations: + - method: GET + path: /summary + - method: POST + path: /reports + resilience: + timeout: 60s +``` + +`GET /summary` is bounded at 5 seconds; `POST /reports` is allowed up to 60 seconds; both inherit the router-level `route_idle_timeout_ms`. + +### Disabling the route timeout for a long-running API + +```yaml +spec: + resilience: + timeout: 0s # no route (response) timeout for this API +``` -- The router gives each upstream connection attempt up to 6 seconds to succeed. -- If the backend is down or unresponsive, requests will fail after approximately 6 seconds instead of hanging indefinitely. \ No newline at end of file +The route will wait indefinitely for the upstream response (bounded only by connect and idle timeouts). Use with care. diff --git a/en/docs/llms.txt b/en/docs/llms.txt index 2c327951f..1713ba373 100644 --- a/en/docs/llms.txt +++ b/en/docs/llms.txt @@ -259,7 +259,7 @@ - [API Gateway Quick Start](https://wso2.com/api-platform/docs/api-gateway/quick-start-guide.md): Configure your first API Proxy using YAML and start routing traffic - [Storage and Backends](https://wso2.com/api-platform/docs/api-gateway/setup/storage-and-backends.md): Configure PostgreSQL as the Gateway Controller storage backend and Redis for distributed rate limiting - [Artifact Templating](https://wso2.com/api-platform/docs/api-gateway/setup/artifact-templating.md): Use Go text template expressions in API Platform Gateway YAML artifacts to inject environment variables and dynamic values at startup -- [Upstream Timeouts](https://wso2.com/api-platform/docs/api-gateway/setup/upstream-timeouts.md): Configure the upstream TCP connection timeout in API Platform Gateway to protect against slow or unreachable backend services +- [Gateway Timeouts](https://wso2.com/api-platform/docs/api-gateway/setup/upstream-timeouts.md): Configure router-level and API-level timeouts (connect, route, idle, and HTTP connection manager) in the API Platform Gateway to protect against slow or unreachable backends and slow clients - [API Gateway Policies Overview](https://wso2.com/api-platform/docs/api-gateway/policies/overview.md): Overview of the policy framework for the standalone gateway - [Writing a Custom Policy](https://wso2.com/api-platform/docs/api-gateway/policies/custom-policies/writing-a-custom-policy.md): Write a custom policy for API Platform Gateway using the Go SDK: implement request and response hooks in the policy chain - [Building Gateway with Custom Policies](https://wso2.com/api-platform/docs/api-gateway/policies/custom-policies/building-gateway-with-custom-policies.md): Use API Platform CLI to bundle custom policies into an API Platform Gateway Docker image for deployment diff --git a/en/mkdocs.yml b/en/mkdocs.yml index 0da6442c6..5727b9786 100644 --- a/en/mkdocs.yml +++ b/en/mkdocs.yml @@ -352,7 +352,7 @@ nav: - Setup: - Storage & Backends: api-gateway/setup/storage-and-backends.md - Artifact Templating: api-gateway/setup/artifact-templating.md - - Upstream Timeouts: api-gateway/setup/upstream-timeouts.md + - Gateway Timeouts: api-gateway/setup/upstream-timeouts.md - Quick Start Guide: api-gateway/quick-start-guide.md - Policies: - Overview: api-gateway/policies/overview.md