Nginx Rolling Deployment Shutdown Auditor

Threat/Impact Level: HIGH | Downtime Risk: HIGH | Time to Fix: 15–30 mins

TL;DR

• What broke: Kubernetes (or your orchestrator) sent SIGTERM to upstream pods during a rolling deploy, but Nginx kept routing requests to those terminating pods before health checks marked them down — resulting in no live upstreams while connecting to upstream and a flood of 502s.
• How to fix it: Add preStop lifecycle hooks to drain connections, set terminationGracePeriodSeconds above your max request duration, and configure Nginx upstream health checks with max_fails/fail_timeout to evict dead peers fast.
• Shortcut: Use our Client-Side Sandbox above to auto-refactor your Nginx config and Deployment YAML — no data leaves your browser.

2024/01/15 03:42:17 [error] 31#31: *18423 no live upstreams while connecting to upstream,
  client: 10.0.1.45, server: api.internal, request: "POST /v1/process HTTP/1.1",
  upstream: "http://backend_pool", host: "api.internal"

How to Fix It (The Solution)

Root Cause Checklist

• Missing preStop lifecycle hook on the upstream container
• terminationGracePeriodSeconds ≤ max request latency
• Nginx upstream block missing max_fails / fail_timeout tuning
• No active health checks (health_check directive — Nginx Plus, or ngx_http_upstream_hc_module)
• Nginx keepalive connections holding sockets open to terminating pods

Basic Fix — Kubernetes Deployment YAML

 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: backend-api
 spec:
   strategy:
     type: RollingUpdate
     rollingUpdate:
-      maxUnavailable: 1
-      maxSurge: 1
+      maxUnavailable: 0
+      maxSurge: 1
   template:
     spec:
+      terminationGracePeriodSeconds: 60
       containers:
       - name: backend
         image: backend-api:v2
         readinessProbe:
           httpGet:
             path: /healthz
             port: 8080
+          initialDelaySeconds: 5
+          periodSeconds: 3
+          failureThreshold: 2
+        lifecycle:
+          preStop:
+            exec:
+              command: ["/bin/sh", "-c", "sleep 15"]

Why sleep 15 in preStop: Kubernetes removes the pod from Endpoints and notifies kube-proxy/Nginx asynchronously. The preStop sleep gives the control plane time to propagate the endpoint removal before the process actually stops accepting connections. 15 seconds covers most cloud provider propagation latency; tune upward if you observe continued 502s.

Enterprise Best Practice — Nginx Upstream Configuration

 upstream backend_pool {
+    zone backend_pool 64k;
+    keepalive 32;
+    keepalive_requests 1000;
+    keepalive_timeout 60s;
 
-    server 10.0.1.10:8080;
-    server 10.0.1.11:8080;
-    server 10.0.1.12:8080;
+    server 10.0.1.10:8080 max_fails=2 fail_timeout=5s;
+    server 10.0.1.11:8080 max_fails=2 fail_timeout=5s;
+    server 10.0.1.12:8080 max_fails=2 fail_timeout=5s;
 }
 
 server {
     location /api/ {
         proxy_pass http://backend_pool;
+        proxy_next_upstream error timeout http_502 http_503;
+        proxy_next_upstream_tries 2;
+        proxy_next_upstream_timeout 10s;
+        proxy_connect_timeout 2s;
+        proxy_read_timeout 30s;
+        proxy_send_timeout 30s;
+        proxy_http_version 1.1;
+        proxy_set_header Connection "";
     }
 }

Critical directives explained:

• max_fails=2 fail_timeout=5s — Nginx marks an upstream down after 2 consecutive failures within 5 seconds, instead of the default 1 fail / 10s. Reduces the blast radius window by 50%.
• proxy_next_upstream error timeout http_502 — Retries the request on a different upstream peer on 502. This is the most impactful single-line fix for end-user impact during deploys.
• proxy_http_version 1.1 + Connection "" — Required for keepalive upstreams. Without this, keepalive is silently disabled and you're creating a new TCP connection per request.
• zone backend_pool 64k — Enables shared memory for upstream state, required for health_check (Nginx Plus) and consistent state across worker processes.

Nginx Plus / OpenResty Active Health Checks

 upstream backend_pool {
     zone backend_pool 64k;
+    # Nginx Plus active health check
+    # For OSS Nginx, use lua-resty-upstream-healthcheck
 
     server 10.0.1.10:8080 max_fails=2 fail_timeout=5s;
     server 10.0.1.11:8080 max_fails=2 fail_timeout=5s;
 }
 
 server {
     location /api/ {
         proxy_pass http://backend_pool;
+        health_check interval=3s fails=2 passes=1 uri=/healthz;
     }
 }

For OSS Nginx, use nginx_upstream_check_module or implement a sidecar health-check exporter that removes pods from the upstream list via the Nginx Plus API or a config reload.

💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.

(For CI/CD pipeline automation, install via npm i -g stackengine).

# policy/require_prestop.rego
package kubernetes.deployment

deny[msg] {
  input.kind == "Deployment"
  container := input.spec.template.spec.containers[_]
  not container.lifecycle.preStop
  msg := sprintf(
    "Container '%v' in Deployment '%v' is missing a preStop lifecycle hook. Rolling deploys will cause upstream drain failures.",
    [container.name, input.metadata.name]
  )
}

deny[msg] {
  input.kind == "Deployment"
  input.spec.template.spec.terminationGracePeriodSeconds < 30
  msg := "terminationGracePeriodSeconds must be >= 30 to allow upstream drain during rolling deploys."
}

conftest test deployment.yaml --policy policy/

checkov -f deployment.yaml --check CKV_K8S_8,CKV_K8S_9
# CKV_K8S_8: Liveness probe configured
# CKV_K8S_9: Readiness probe configured

{
  "$schema": "http://json-schema.org/draft-07/schema",
  "properties": {
    "terminationGracePeriodSeconds": {
      "type": "integer",
      "minimum": 30,
      "description": "Must exceed max request duration to prevent upstream drain 502s"
    }
  },
  "required": ["terminationGracePeriodSeconds"]
}

# Using k6
k6 run --duration 30s --vus 50 smoke_test.js
# Assert: http_req_failed rate == 0

kubectl rollout undo deployment/backend-api

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-graceful-shutdown
spec:
  validationFailureAction: enforce
  rules:
  - name: check-prestop-hook
    match:
      resources:
        kinds: [Deployment]
    validate:
      message: "preStop hook required on all containers to prevent Nginx upstream 502s during rolling deploys."
      pattern:
        spec:
          template:
            spec:
              containers:
              - lifecycle:
                  preStop:
                    exec:
                      command: "?*"