Nginx Ingress Readiness 503 Rollout Inspector

The Incident (What Does the Error Mean?)

Raw event output from kubectl describe pod -n ingress-nginx <controller-pod>:

Warning  Unhealthy  3s    kubelet  Readiness probe failed:
HTTP probe failed with statuscode: 503
GET http://10.244.1.47:10254/healthz

Kubernetes hit the readiness probe endpoint and got back 503 Service Unavailable instead of 200 OK. The pod is alive (liveness hasn't fired yet) but Kubernetes will not route traffic to it. If this affects all replicas simultaneously during a rolling update, your ingress layer goes completely dark — every Ingress resource in the cluster stops serving traffic. This is not a graceful degradation. It is a full ingress outage.

The Attack Vector / Blast Radius

This failure cascades fast:

1. Rollout deadlock: With maxUnavailable: 0 (default in many hardened configs), the rollout waits indefinitely for the new pod to become Ready. It never does. The old pods are still running but the rollout is frozen — your CI/CD pipeline hangs, alerts fire, and engineers start force-killing things.
2. Full ingress blackout: If old pods are already terminated (aggressive terminationGracePeriodSeconds or manual drain), and new pods are stuck in 0/1 Ready, zero ingress-nginx pods are in the Service endpoints slice. Every HTTP/HTTPS request to your cluster returns a connection refused or LB 502.
3. Cascading cert-manager failures: cert-manager HTTP-01 ACME challenges route through ingress-nginx. A 503 readiness failure during certificate renewal silently breaks TLS renewal — you find out 30 days later when certs expire.
4. Root causes ranked by frequency:
   • Port mismatch between probe definition and --healthz-port arg (most common after Helm chart version bumps)
   • Upstream backend not ready — the controller is healthy but a default backend Service has no Endpoints
   • externalTrafficPolicy: Local — node has no local endpoints, kube-proxy drops the probe
   • Resource starvation — OOMKilled nginx worker, controller returns 503 on /healthz while restarting
   • Webhook misconfiguration — ValidatingWebhookConfiguration rejecting ingress objects, controller enters error state

Prevention in CI/CD

1. Helm Chart Lint + Conftest Policy

Add an OPA/Conftest policy that enforces readiness probe port matches --healthz-port arg:

# policy/ingress_probe.rego
package ingress_nginx

deny[msg] {
  container := input.spec.template.spec.containers[_]
  container.name == "controller"
  probe_port := container.readinessProbe.httpGet.port
  healthz_arg := [a | a := container.args[_]; startswith(a, "--healthz-port=")][0]
  expected_port := to_number(split(healthz_arg, "=")[1])
  probe_port != expected_port
  msg := sprintf("Readiness probe port %v does not match --healthz-port %v", [probe_port, expected_port])
}

# In CI pipeline
helm template ingress-nginx ingress-nginx/ingress-nginx -f values.yaml | \
  conftest test - --policy policy/

2. Checkov Static Analysis

checkov -f controller-deployment.yaml \
  --check CKV_K8S_8  # readiness probe defined
  --check CKV_K8S_9  # liveness probe defined

3. Pre-Rollout Smoke Test in ArgoCD / Flux

Add a PreSync Job or ArgoCD Resource Hook that validates the default backend is healthy before the controller rolls:

apiVersion: batch/v1
kind: Job
metadata:
  name: pre-rollout-backend-check
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
      - name: check
        image: curlimages/curl:8.7.1
        command:
          - sh
          - -c
          - |
            curl -sf http://ingress-nginx-defaultbackend.ingress-nginx.svc.cluster.local:8080/healthz \
              || (echo "Default backend not ready. Blocking rollout."; exit 1)
      restartPolicy: Never

4. Monitor with Prometheus Alert

# PrometheusRule
- alert: IngressNginxControllerNotReady
  expr: kube_pod_container_status_ready{namespace="ingress-nginx",container="controller"} == 0
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "All ingress-nginx controller pods are NotReady — cluster ingress is down"