Nginx K8s Ingress Long-Running API 504

The Incident (What Does the Error Mean?)

The raw error your client or upstream proxy receives:

HTTP/1.1 504 Gateway Time-out
Server: nginx/1.25.3
Date: Mon, 01 Jul 2024 14:32:07 GMT
Content-Type: text/html
Content-Length: 167

<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.25.3</center>
</body>
</html>

And in the Nginx Ingress controller pod logs:

2024/07/01 14:32:07 [error] 31#31: *10482 upstream timed out (110: Connection timed out)
while reading response header from upstream, client: 10.0.1.45,
server: api.example.com, request: "POST /api/v1/batch-process HTTP/1.1",
upstream: "http://10.96.14.22:8080/api/v1/batch-process",
host: "api.example.com"

Immediate consequence: Every API call that takes longer than 60 seconds (the Nginx default) is hard-killed at the proxy layer. Your upstream pod is still processing — it has no idea the client is gone. This means orphaned goroutines/threads on the backend, wasted compute, and a client that must retry from scratch.

The Attack Vector / Blast Radius

This is a cascading availability failure, not a one-shot timeout:

1. Client retries amplify backend load. A frustrated client (or an SDK with automatic retry logic) re-submits the same expensive request. Your backend now has two in-flight copies of the same job. Multiply by N users and you have a thundering herd.
2. Orphaned upstream connections exhaust worker pools. The upstream pod never received a RST — it keeps the connection open and burns a worker thread until its own idle timeout fires. Under load, this drains your Gunicorn/uWSGI/Tomcat worker pool entirely, causing subsequent fast requests to also queue and timeout.
3. HPA fires on CPU/memory, not on the real cause. Horizontal Pod Autoscaler sees high resource usage from the orphaned workers and spins up more pods. Each new pod inherits the same misconfigured Ingress. Scaling does nothing; you burn money and still serve 504s.
4. Data integrity risk on non-idempotent endpoints. If your batch job writes to a database mid-flight when the proxy kills the connection, you may get partial writes. Without proper transaction rollback, this is a silent data corruption vector.
5. SLO breach. A 504 is logged as a server-side error. Your error budget drains. If you have an alerting threshold on 5xx rate, this triggers PagerDuty at 3am.

The blast radius is not just one endpoint — a single misconfigured Ingress resource with pathType: Prefix on / covers every route in that virtual host.

Prevention in CI/CD

1. OPA/Gatekeeper Policy — Enforce Timeout Annotations on Long-Running Path Ingresses

Tag Ingress resources that serve async/batch routes with a label, then enforce the annotation contract:

# opa/policies/ingress-timeout-policy.rego
package kubernetes.admission

deny[msg] {
  input.request.kind.kind == "Ingress"
  input.request.object.metadata.labels["route-type"] == "long-running"
  not input.request.object.metadata.annotations["nginx.ingress.kubernetes.io/proxy-read-timeout"]
  msg := "Long-running Ingress routes MUST declare nginx.ingress.kubernetes.io/proxy-read-timeout"
}

deny[msg] {
  input.request.kind.kind == "Ingress"
  input.request.object.metadata.labels["route-type"] == "long-running"
  timeout := to_number(input.request.object.metadata.annotations["nginx.ingress.kubernetes.io/proxy-read-timeout"])
  timeout < 120
  msg := sprintf("proxy-read-timeout value %v is below minimum 120s for long-running routes", [timeout])
}

2. Checkov — Static Analysis in Pull Requests

Add a custom Checkov check to your CI pipeline:

# .github/workflows/k8s-lint.yml
- name: Run Checkov on Kubernetes manifests
  uses: bridgecrewio/checkov-action@master
  with:
    directory: k8s/
    framework: kubernetes
    check: CKV_K8S_INGRESS_TIMEOUT   # custom check ID
    soft_fail: false

3. Helm Values Schema Validation

If you manage Ingress via Helm, enforce timeout values in values.schema.json:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "properties": {
    "ingress": {
      "properties": {
        "proxyReadTimeout": {
          "type": "integer",
          "minimum": 60,
          "maximum": 600,
          "description": "Nginx proxy-read-timeout in seconds. Must be set explicitly."
        }
      },
      "required": ["proxyReadTimeout"]
    }
  }
}

4. Synthetic Monitoring Canary

Deploy a Prometheus Blackbox Exporter probe that fires a synthetic long-running request (e.g., a /healthz/slow?delay=90 endpoint on your service) every 5 minutes. Alert if it returns 504:

# prometheus/rules/ingress-timeout.yml
groups:
- name: ingress.timeout
  rules:
  - alert: LongRunningEndpoint504
    expr: probe_http_status_code{job="blackbox-batch-api"} == 504
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "504 on long-running API endpoint — check Ingress proxy-read-timeout"

This catches timeout regressions the moment a new Ingress manifest is deployed, before real users hit it.