How to Fix Nginx 'upstream timed out (110: Connection timed out)' While Reading Response Header
Threat/Impact Level: HIGH | Exploitability/Downtime Risk: HIGH | Time to Fix: 15–45 mins depending on root cause layer
TL;DR
- What broke: Nginx waited 60 seconds for the first byte of an HTTP response header from
http://backend:8080and got nothing. The OS returnedETIMEDOUT (110), Nginx logged the error, and your users got a 504 Gateway Timeout. - How to fix it: Identify whether the timeout is in Nginx config (too short), the backend app (slow query, thread starvation, GC pause), or the network layer (DNS, container networking, firewall). Then tune
proxy_read_timeout, fix the upstream bottleneck, and add upstream health checks. - Shortcut: Use our Client-Side Sandbox above to paste your Nginx config — it will auto-diagnose the timeout chain and output a refactored config with correct timeout values and health check directives.
▼ 🔍 Root Cause Breakdown & Kernel Diagnostics [Click to Expand Deep Dive]
The Incident (What Does This Error Mean?)
Raw log output:
2024/01/15 03:47:22 [error] 31#31: *18423 upstream timed out (110: Connection timed out)
while reading response header from upstream 'http://backend:8080/api/v2/process',
client: 10.0.1.45, server: api.example.com, request: "POST /api/v2/process HTTP/1.1",
upstream: "http://172.17.0.3:8080/api/v2/process", host: "api.example.com"What this means precisely: Nginx successfully opened a TCP connection to the backend and forwarded the full request. The backend accepted the connection and received the data — but never sent back even the first byte of the HTTP response header within the proxy_read_timeout window (default or configured: 60s). This is not a connection refusal. The backend is alive but not responding in time.
Immediate consequence: Nginx closes the upstream socket, returns HTTP 504 Gateway Timeout to the client, and logs this error. If you have multiple upstream workers in a pool, Nginx may retry on the next upstream — multiplying backend load during an already-degraded state.
▼ 🔍 Detailed Impact & Blast Radius Analysis [Click to Expand Deep Dive]
The Attack Vector / Blast Radius
This is a cascading failure scenario, not a one-off timeout. Here is the failure chain:
Backend thread/goroutine pool exhaustion: One slow database query holds a thread. New requests queue. Queue depth exceeds timeout window. All requests start timing out. Nginx retries amplify backend load by
proxy_next_upstreamfactor. You now have a thundering herd.Connection pool starvation: If your backend uses a DB connection pool (HikariCP, pgBouncer, etc.) and pool is saturated, every new request blocks waiting for a connection. 60 seconds passes. Nginx kills it. The backend thread is still blocked — it doesn't know Nginx gave up. You now have zombie threads holding DB connections.
Nginx worker process saturation: Each timed-out request held an Nginx worker connection for the full 60s. At 1024 worker connections and 60s timeout, you can sustain only ~17 req/s before Nginx itself saturates. Your proxy layer becomes the bottleneck, not just the backend.
Security angle — slowloris-variant against your own stack: A misconfigured
proxy_read_timeoutthat is too long (e.g., 300s) means a slow backend (or a deliberately slow attacker-controlled internal service in a compromised environment) can hold Nginx workers open for 5 minutes each. This is effectively a resource exhaustion vector from within your own cluster.
Blast radius: Full service degradation within 2–3 minutes of initial backend slowdown. All downstream clients see 504s. Upstream retries amplify backend load. Recovery requires backend restart AND Nginx connection drain.
How to Fix It (The Solution)
Step 0: Confirm Which Layer Is Timing Out
Before touching config, run this from inside the Nginx container/pod:
# Test raw TCP connect time to backend
curl -v --max-time 5 http://backend:8080/healthz
# If curl also hangs, the problem is the backend, not Nginx timeout config
# If curl succeeds fast, the problem is specific to your slow endpoint
# Check backend thread/connection state
curl http://backend:8080/actuator/metrics/jvm.threads.live # Spring Boot
curl http://backend:8080/debug/pprof/goroutine?debug=1 # GoBasic Fix — Tune Nginx Timeout Values
Only do this if your backend legitimately needs more time (e.g., long-running export jobs). This is a band-aid, not a fix.
http {
upstream backend_pool {
server backend:8080;
+ keepalive 32;
+ keepalive_requests 1000;
+ keepalive_timeout 75s;
}
server {
location /api/ {
proxy_pass http://backend_pool;
- # Default or previously missing — Nginx falls back to 60s
+ proxy_connect_timeout 10s;
+ proxy_send_timeout 60s;
+ proxy_read_timeout 120s; # Increase ONLY if backend genuinely needs it
+ proxy_http_version 1.1;
+ proxy_set_header Connection "";
}
}
}Enterprise Best Practice — Upstream Health Checks, Circuit Breaking, and Per-Route Timeouts
http {
upstream backend_pool {
zone backend_zone 64k;
server backend:8080;
- # No health checks — dead backends stay in rotation
+ keepalive 32;
+ keepalive_requests 1000;
+ keepalive_timeout 75s;
}
+ # Separate upstream for long-running async endpoints
+ upstream backend_async_pool {
+ zone backend_async_zone 64k;
+ server backend:8080;
+ keepalive 8;
+ }
server {
# Fast API endpoints — strict timeout
location /api/v2/ {
proxy_pass http://backend_pool;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
- proxy_read_timeout 60s;
+ proxy_connect_timeout 5s;
+ proxy_send_timeout 30s;
+ proxy_read_timeout 30s;
+ # Circuit-break: don't retry on timeout — avoids thundering herd
+ proxy_next_upstream error timeout http_502 http_503;
+ proxy_next_upstream_tries 2;
+ proxy_next_upstream_timeout 10s;
+ # Return 504 immediately with a clean error body
+ proxy_intercept_errors on;
+ error_page 504 /errors/504.json;
}
+ # Long-running jobs get their own location with relaxed timeout
+ location /api/v2/export {
+ proxy_pass http://backend_async_pool;
+ proxy_http_version 1.1;
+ proxy_set_header Connection "";
+ proxy_connect_timeout 5s;
+ proxy_send_timeout 30s;
+ proxy_read_timeout 300s; # Explicit, documented, scoped
+ proxy_next_upstream off; # No retry for idempotency-unsafe ops
+ }
}
}For Kubernetes (nginx-ingress annotations):
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: backend-ingress
annotations:
nginx.ingress.kubernetes.io/proxy-connect-timeout: "5"
- # Missing — inherits 60s default
+ nginx.ingress.kubernetes.io/proxy-send-timeout: "30"
+ nginx.ingress.kubernetes.io/proxy-read-timeout: "30"
+ nginx.ingress.kubernetes.io/proxy-next-upstream: "error timeout"
+ nginx.ingress.kubernetes.io/proxy-next-upstream-tries: "2"
+ # Enable upstream keepalive
+ nginx.ingress.kubernetes.io/upstream-keepalive-connections: "32"
+ nginx.ingress.kubernetes.io/upstream-keepalive-timeout: "75"Fix the backend — this is the real fix:
# Spring Boot application.properties
-# No connection pool config — using HikariCP defaults
+spring.datasource.hikari.maximum-pool-size=20
+spring.datasource.hikari.connection-timeout=3000
+spring.datasource.hikari.idle-timeout=300000
+spring.datasource.hikari.max-lifetime=1200000
+# Add slow query logging to find the actual culprit
+spring.datasource.hikari.leak-detection-threshold=30000💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.
(For CI/CD pipeline automation, install via
npm i -g stackengine).
▼ 🔍 Continuous Pipeline Governance & CI/CD Guardrails [Click to Expand Deep Dive]
Prevention in CI/CD
1. Lint Nginx Config in Pipeline
# In your Dockerfile or CI step — catch missing timeout directives at build time
nginx -t -c /etc/nginx/nginx.conf
# Use gixy (Nginx static analyzer) for security and config issues
pip install gixy
gixy /etc/nginx/nginx.conf2. OPA/Conftest Policy — Enforce Timeout Directives
# policy/nginx_timeouts.rego
package nginx
deny[msg] {
input.http.server.location[_].proxy_read_timeout == null
msg := "POLICY VIOLATION: proxy_read_timeout must be explicitly set on all proxy locations. Default 60s is insufficient for production."
}
deny[msg] {
timeout := input.http.server.location[_].proxy_read_timeout
to_number(trim(timeout, "s")) > 120
msg := sprintf("POLICY VIOLATION: proxy_read_timeout of %v exceeds 120s maximum. Use a dedicated async upstream for long-running operations.", [timeout])
}3. Load Test Timeout Boundaries in Staging
# k6 script — validate that 504s appear at expected threshold
import http from 'k6/http';
import { check } from 'k6';
export let options = {
scenarios: {
timeout_boundary: {
executor: 'constant-vus',
vus: 50,
duration: '120s',
},
},
thresholds: {
// Fail the pipeline if more than 1% of requests timeout
'http_req_failed{status:504}': ['rate<0.01'],
'http_req_duration{status:200}': ['p(99)<25000'],
},
};
export default function () {
let res = http.post('https://staging.api.example.com/api/v2/process',
JSON.stringify({ payload: 'test' }),
{ headers: { 'Content-Type': 'application/json' }, timeout: '35s' }
);
check(res, { 'not a 504': (r) => r.status !== 504 });
}4. Alerting — Don't Wait for Users to Report It
# Prometheus alerting rule
groups:
- name: nginx_upstream_timeouts
rules:
- alert: NginxUpstreamTimeoutRateHigh
expr: |
rate(nginx_upstream_response_time_seconds_count{status="504"}[5m]) /
rate(nginx_upstream_response_time_seconds_count[5m]) > 0.05
for: 2m
labels:
severity: critical
annotations:
summary: "Nginx upstream timeout rate exceeds 5% — backend degraded"
runbook: "https://wiki.internal/runbooks/nginx-504"Official Grounding & Normative Specifications
The technical diagnostics, error recovery protocols, and prevention patterns in this specification are grounded in official vendor documentation and standards:
Official runtime configuration documentation governing lock acquisition timeouts, deadlock detection cycles, and wait logging.
Federal security guidelines for secret exfiltration prevention, automated integrity checks, and peer sign-offs.
Core memory subsystem controls, pressure stall information (PSI), and kernel OOM invocation mechanics.
Industry-standard specification for container execution boundaries and POSIX namespace isolation.