Nginx 504 Gateway Timeout PHP-FPM Inspector

The Incident (What Does the Error Mean?)

This is what hits your Nginx error log:

2024/07/15 03:42:17 [error] 18#18: *4821 upstream timed out (110: Connection timed out)
  while connecting to upstream, client: 203.0.113.45, server: app.example.com,
  request: "POST /api/checkout HTTP/1.1",
  upstream: "fastcgi://unix:/run/php/php8.2-fpm.sock",
  host: "app.example.com"

Immediate consequence: Every request hitting that upstream returns a blank 504 to the end user. If this is a checkout, API, or auth endpoint, you are losing revenue and sessions in real time. Nginx did not crash — it is healthy. PHP-FPM is the bottleneck. The socket or TCP port exists, but no worker picked up the connection within fastcgi_connect_timeout, or a worker accepted it but did not respond within fastcgi_read_timeout.

Two distinct failure modes produce this identical error:

Distinguish them before touching any config.

The Attack Vector / Blast Radius

This is a cascading saturation failure, not an isolated slow request.

1. Worker pool exhaustion: One slow query (e.g., a missing DB index, a synchronous third-party API call with no timeout) holds a PHP-FPM worker for 30–120 seconds. With pm.max_children = 5 (the Debian/Ubuntu default), five concurrent slow requests drain the entire pool. Request #6 hits the socket, gets EAGAIN or a full listen backlog, and Nginx times out at fastcgi_connect_timeout (default: 60s). The 504 storm begins.

2. OOM-triggered worker recycling: If pm.max_requests is unset and a memory leak exists in your application, workers balloon in RSS. The kernel OOM killer terminates FPM workers mid-request. Nginx sees a broken pipe and logs the same 110 error.

3. Unix socket backlog overflow: The default listen.backlog = 511 in FPM is often overridden to -1 (OS default, typically 128 on Linux) by misconfigured pool files. Under burst traffic, the kernel drops new connection attempts silently. Nginx gets ETIMEDOUT (110) instead of ECONNREFUSED (111), making diagnosis harder.

4. Security implication: A single unauthenticated endpoint that triggers an expensive operation (report generation, image processing, recursive DB query) becomes a low-effort application-layer DoS vector. An attacker sending 10 concurrent POST requests to /export/csv can pin all FPM workers and take down the entire vhost — including your login page.

Prevention in CI/CD

1. Lint FPM pool configs in your pipeline

# .github/workflows/config-lint.yml
- name: Validate PHP-FPM pool config
  run: |
    php-fpm8.2 --test -y ./deploy/php-fpm/www.conf
    # Fail if pm.max_children is below threshold
    MAX=$(grep 'pm.max_children' ./deploy/php-fpm/www.conf | awk -F= '{print $2}' | tr -d ' ')
    [ "$MAX" -ge 20 ] || (echo "pm.max_children too low: $MAX" && exit 1)

2. Checkov / OPA policy for Nginx timeout hygiene

# checkov custom check: ensure fastcgi_read_timeout is set and >= 30s
import re
from checkov.common.models.enums import CheckResult

def check_fastcgi_timeout(config_content: str) -> CheckResult:
    match = re.search(r'fastcgi_read_timeout\s+(\d+)', config_content)
    if not match or int(match.group(1)) < 30:
        return CheckResult.FAILED
    return CheckResult.PASSED

3. Prometheus + Alertmanager — alert before pool exhaustion

# prometheus alert: fire at 80% worker saturation, not at 100%
- alert: PHPFPMWorkerSaturation
  expr: |
    (phpfpm_active_processes / phpfpm_max_children) > 0.80
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "PHP-FPM pool {{ $labels.pool }} at {{ $value | humanizePercentage }} capacity"
    runbook: "https://wiki.internal/runbooks/php-fpm-504"

- alert: PHPFPMMaxChildrenReached
  expr: phpfpm_max_children_reached_total > 0
  for: 1m
  labels:
    severity: critical

Use php-fpm_exporter to expose /status to Prometheus. The max children reached counter is the single most important signal — it increments every time FPM had to reject a new worker spawn, which is the direct precursor to a 504 storm.

4. Load test before deploy

# Use k6 to reproduce the saturation condition in staging
k6 run --vus 60 --duration 30s - <<'EOF'
import http from 'k6/http';
export default function () {
  http.post('https://staging.example.com/api/checkout', JSON.stringify({test: true}), {
    headers: { 'Content-Type': 'application/json' },
    timeout: '10s',
  });
}
EOF
# If p99 latency spikes and error rate > 1%, your pool config is undersized for prod traffic.

5. Terraform / Ansible: enforce config-as-code for FPM pool values

# terraform variable validation — prevent deploying with default pool sizes
variable "fpm_max_children" {
  type = number
  validation {
    condition     = var.fpm_max_children >= 20
    error_message = "pm.max_children must be >= 20. Default of 5 causes 504s under any real load."
  }
}