Nginx No Live Upstreams Health-Check Auditor

Threat/Impact Level: CRITICAL | Downtime Risk: HIGH | Time to Fix: 15–45 mins depending on backend state

TL;DR

• What broke: Every server in the Nginx upstream pool simultaneously failed passive or active health checks, leaving max_fails thresholds breached and fail_timeout windows active — Nginx has literally no server to route to and returns 502 Bad Gateway to every incoming request.
• How to fix it: Tune max_fails / fail_timeout to prevent premature ejection, add a backup server as a circuit-breaker, enable Nginx Plus active health checks (or ngx_http_upstream_check_module on OSS), and fix the root cause killing your backends (OOM, DB connection exhaustion, misconfigured readiness probes).
• Shortcut: Use our Client-Side Sandbox above to auto-refactor your upstream block — paste your config, get corrected code without leaking your internal IPs or hostnames.

2024/01/15 03:42:17 [error] 19#19: *58423 no live upstreams while connecting to upstream,
  client: 10.0.1.45, server: api.internal, request: "POST /api/v2/orders HTTP/1.1",
  upstream: "http://backend_pool/api/v2/orders",
  host: "api.internal"

How to Fix It

Basic Fix — Stop Premature Backend Ejection

The most common root cause is max_fails being too low and fail_timeout being too short or too long for your traffic pattern.

upstream backend_pool {
    # Round-robin is default, no directive needed

-   server 10.0.1.10:8080;
-   server 10.0.1.11:8080;
-   server 10.0.1.12:8080;
+   server 10.0.1.10:8080 max_fails=3 fail_timeout=30s;
+   server 10.0.1.11:8080 max_fails=3 fail_timeout=30s;
+   server 10.0.1.12:8080 max_fails=3 fail_timeout=30s;
+
+   # Backup: static maintenance page server — ALWAYS have one
+   server 10.0.1.99:8080 backup;
+
+   keepalive 32;
}

server {
    location /api/ {
        proxy_pass http://backend_pool;
-       proxy_connect_timeout 5s;
-       proxy_read_timeout 10s;
+       proxy_connect_timeout 3s;
+       proxy_read_timeout 30s;
+       proxy_next_upstream error timeout http_502 http_503;
+       proxy_next_upstream_tries 3;
+       proxy_next_upstream_timeout 10s;
    }
}

Key changes explained:

• max_fails=3 — requires 3 consecutive failures before ejection, not 1.
• fail_timeout=30s — backend is penalized for 30s, but also only counts failures within a 30s window. Tune this to your backend's expected recovery time.
• backup server — Nginx only routes here when ALL primary servers are down. This is your 502 escape hatch.
• proxy_next_upstream — on a 502/503 from one backend, Nginx retries the next peer before giving up.

Enterprise Best Practice — Active Health Checks + Observability

Passive health checks are reactive. You need active checks that probe backends before Nginx routes live traffic to them.

Nginx Plus (commercial) — active health checks:

upstream backend_pool {
    zone backend_pool 64k;  # Required for active health checks and status API

-   server 10.0.1.10:8080 max_fails=1 fail_timeout=10s;
-   server 10.0.1.11:8080 max_fails=1 fail_timeout=10s;
-   server 10.0.1.12:8080 max_fails=1 fail_timeout=10s;
+   server 10.0.1.10:8080 max_fails=3 fail_timeout=30s slow_start=20s;
+   server 10.0.1.11:8080 max_fails=3 fail_timeout=30s slow_start=20s;
+   server 10.0.1.12:8080 max_fails=3 fail_timeout=30s slow_start=20s;
+   server 10.0.1.99:8080 backup;
+
+   keepalive 64;
+   keepalive_requests 1000;
+   keepalive_timeout 75s;
}

server {
    location /api/ {
        proxy_pass http://backend_pool;
+       health_check interval=5s fails=3 passes=2 uri=/health match=api_healthy;
+       proxy_next_upstream error timeout http_502 http_503 http_504;
+       proxy_next_upstream_tries 2;
    }
}

+# Health check response validation
+match api_healthy {
+    status 200;
+    header Content-Type ~ "application/json";
+    body ~ '"status":"ok"';
+}

OSS Nginx with ngx_http_upstream_check_module (Tengine/compiled):

upstream backend_pool {
-   server 10.0.1.10:8080;
-   server 10.0.1.11:8080;
+   server 10.0.1.10:8080;
+   server 10.0.1.11:8080;
+   server 10.0.1.12:8080;
+   server 10.0.1.99:8080 backup;
+
+   check interval=3000 rise=2 fall=3 timeout=1000 type=http;
+   check_http_send "GET /health HTTP/1.0\r\nHost: backend\r\n\r\n";
+   check_http_expect_alive http_2xx;
}

slow_start is critical for rolling restarts. Without it, a freshly restarted backend gets slammed with full traffic immediately, often causing it to fail health checks again under the initial load spike — re-triggering the exact problem you just fixed.

💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.

(For CI/CD pipeline automation, install via npm i -g stackengine).

# In your CI step — catches syntax errors and missing directives
nginx -t -c /etc/nginx/nginx.conf

# Or with gixy (Nginx security linter)
pip install gixy
gixy /etc/nginx/nginx.conf

# .github/workflows/nginx-validate.yml
- name: Validate upstream health endpoint matches readiness probe
  run: |
    HEALTH_URI=$(grep 'health_check.*uri=' nginx/upstream.conf | grep -oP 'uri=\K[^ ;]+')
    READINESS_PATH=$(kubectl get deployment api -o jsonpath='{.spec.template.spec.containers[0].readinessProbe.httpGet.path}')
    if [ "$HEALTH_URI" != "$READINESS_PATH" ]; then
      echo "MISMATCH: Nginx health_check URI ($HEALTH_URI) != K8s readiness probe ($READINESS_PATH)"
      exit 1
    fi

# policy/nginx_upstream.rego
package nginx.upstream

deny[msg] {
    upstream := input.upstreams[_]
    not has_backup_server(upstream)
    msg := sprintf("Upstream '%v' has no backup server defined. All-backends-down scenario has no fallback.", [upstream.name])
}

deny[msg] {
    server := input.upstreams[_].servers[_]
    server.max_fails < 2
    msg := sprintf("Server '%v' has max_fails < 2. Single transient failure will eject backend from rotation.", [server.address])
}

has_backup_server(upstream) {
    server := upstream.servers[_]
    server.backup == true
}

# Prometheus alert — trigger when >50% of upstreams are down, not 100%
- alert: NginxUpstreamPoolDegraded
  expr: |
    (nginx_upstream_peers_down / nginx_upstream_peers_total) > 0.5
  for: 30s
  labels:
    severity: critical
  annotations:
    summary: "Upstream pool {{ $labels.upstream }} is {{ $value | humanizePercentage }} down — imminent no-live-upstreams failure"