Nginx Least_Conn All Unhealthy Upstream Audit

Threat/Impact Level: HIGH | Downtime Risk: HIGH | Time to Fix: 15–30 mins

TL;DR

• What broke: Every peer in your least_conn upstream pool hit max_fails within fail_timeout, so Nginx has zero live peers to route to — every inbound request dies with a 502.
• How to fix it: Tune max_fails/fail_timeout, add a backup server or keepalive, and implement active health checks so Nginx stops marking peers dead on transient errors.
• Use the Client-Side Sandbox above to paste your upstream block and auto-refactor it with corrected health-check parameters.

2024/01/15 03:42:17 [error] 1234#1234: *98765 no live upstreams while connecting to upstream,
  client: 10.0.1.45, server: api.example.com, request: "POST /v1/orders HTTP/1.1",
  upstream: "http://backend_pool/v1/orders", host: "api.example.com"

How to Fix It

Basic Fix — Tune Passive Health Check Thresholds

 upstream backend_pool {
     least_conn;
 
-    server 10.0.0.1:8080;
-    server 10.0.0.2:8080;
-    server 10.0.0.3:8080;
+    server 10.0.0.1:8080 max_fails=3 fail_timeout=30s;
+    server 10.0.0.2:8080 max_fails=3 fail_timeout=30s;
+    server 10.0.0.3:8080 max_fails=3 fail_timeout=30s;
+
+    # Backup fires only when ALL primaries are down
+    server 10.0.0.99:8080 backup;
 }

Why: Raising max_fails to 3 means a single timeout during a deploy doesn't eject the peer. fail_timeout=30s controls both the window for counting failures AND the cooldown before re-admission — tune this to be shorter than your deploy window.

Enterprise Best Practice — Active Health Checks + Keepalive (Nginx Plus or OSS + upstream_check module)

 upstream backend_pool {
     least_conn;
 
-    server 10.0.0.1:8080 max_fails=1 fail_timeout=10s;
-    server 10.0.0.2:8080 max_fails=1 fail_timeout=10s;
-    server 10.0.0.3:8080 max_fails=1 fail_timeout=10s;
+    server 10.0.0.1:8080 max_fails=3 fail_timeout=20s;
+    server 10.0.0.2:8080 max_fails=3 fail_timeout=20s;
+    server 10.0.0.3:8080 max_fails=3 fail_timeout=20s;
+    server 10.0.0.99:8080 backup;
+
+    keepalive 32;
+    keepalive_requests 1000;
+    keepalive_timeout 60s;
+
+    # Nginx Plus active health check (remove if OSS)
+    zone backend_pool 64k;
 }
 
 server {
     location /api/ {
         proxy_pass http://backend_pool;
+        proxy_next_upstream error timeout http_502 http_503;
+        proxy_next_upstream_tries 2;
+        proxy_next_upstream_timeout 5s;
+        proxy_connect_timeout 3s;
+        proxy_read_timeout 30s;
+
+        # Nginx Plus only:
+        # health_check interval=5s fails=2 passes=3 uri=/healthz;
     }
 }

Critical additions:

• proxy_next_upstream http_502 http_503 — on a bad response, Nginx tries the next peer before failing the request. This alone prevents the spiral in many cases.
• proxy_next_upstream_tries 2 — caps retry attempts so you don't amplify load.
• zone directive (Nginx Plus) — enables shared memory for health state across workers, required for active health_check.
• backup server — a static or minimal-capacity node that absorbs traffic only when all primaries are ejected. Can point to a maintenance-mode app returning 503 with Retry-After headers instead of a raw 502.

💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.

(For CI/CD pipeline automation, install via npm i -g stackengine).

# Use gixy (Nginx config static analyzer)
pip install gixy
gixy /etc/nginx/nginx.conf
# Catches missing proxy_next_upstream, zero backup peers, etc.

# policy/nginx_upstream.rego
package nginx.upstream

deny[msg] {
    block := input.upstreams[_]
    block.directive == "upstream"
    not has_backup(block)
    msg := sprintf("Upstream '%v' has no backup server defined", [block.name])
}

has_backup(block) {
    server := block.servers[_]
    server.params[_] == "backup"
}

# Simulate all backends going dark for 15s using tc or iptables
for host in 10.0.0.1 10.0.0.2 10.0.0.3; do
  ssh $host "sudo iptables -A INPUT -p tcp --dport 8080 -j DROP &
             sleep 15 &&
             sudo iptables -D INPUT -p tcp --dport 8080 -j DROP" &
done
# Monitor: watch -n1 'curl -o /dev/null -sw "%{http_code}" https://api.example.com/healthz'

# Prometheus alert (requires nginx-prometheus-exporter)
- alert: NginxUpstreamPeersLow
  expr: nginx_upstream_peers_up{upstream="backend_pool"} < 2
  for: 30s
  labels:
    severity: critical
  annotations:
    summary: "Upstream backend_pool has fewer than 2 live peers"