Nginx PHP-FPM Pool Exhaustion 111 Refused

The Incident (What Does the Error Mean?)

Your Nginx error log is printing this:

2024/01/15 03:42:17 [error] 18#18: *58231 connect() failed (111: Connection refused)
  while connecting to upstream,
  client: 203.0.113.47, server: example.com,
  request: "POST /api/checkout HTTP/1.1",
  upstream: "http://127.0.0.1:9000",
  host: "example.com"

Error 111 (ECONNREFUSED) means the TCP connection was actively refused — the port is bound but nothing is accepting the connection, or the listen backlog queue is overflowing and the kernel is dropping SYNs.

PHP-FPM's process manager has hit pm.max_children. Every worker slot is occupied by a running (or hung) PHP request. The socket/port has no capacity to accept the next connection. Nginx has nowhere to send the request and immediately 502s the user.

Immediate consequence: 100% of PHP-dependent traffic returns HTTP 502. Static assets served directly by Nginx still work, which is why your homepage may load but every dynamic request fails.

The Attack Vector / Blast Radius

Pool exhaustion is a cascading failure, not a single-point event:

1. Slow upstream dependency (database, Redis, external API) causes PHP workers to block waiting for I/O. Response times climb from 200ms to 8s+.
2. New requests queue behind the blocked workers. pm.max_children is hit. The FPM listen backlog (listen.backlog) fills.
3. Nginx upstream timeout (fastcgi_read_timeout) hasn't fired yet, so Nginx keeps the connection open, consuming worker slots on the Nginx side too.
4. The kernel starts refusing new TCP connections to port 9000 / the UNIX socket — this is the 111: Connection refused you see.
5. Nginx logs the error, returns 502. If you have no circuit breaker, every retry from the client hammers the already-exhausted pool harder.
6. OOM risk: If you attempt to fix this in-flight by restarting PHP-FPM under load, the master process spawns pm.start_servers children simultaneously, potentially spiking RAM and triggering the OOM killer on the PHP-FPM master itself.

Secondary blast radius: Any health check endpoint backed by PHP will also fail, causing your load balancer to mark the instance unhealthy and potentially drain it from the pool — turning a performance incident into a full availability incident.

Prevention in CI/CD

1. Load test before deploy — catch exhaustion in staging

# k6 smoke test — run in CI pipeline before promoting to prod
k6 run --vus 60 --duration 30s load-test.js
# Fail the pipeline if p95 latency > 2s or error rate > 1%

2. Ansible / Terraform — enforce `pm.max_children` is calculated, not hardcoded

# terraform/modules/php_fpm/variables.tf
variable "worker_ram_mb" { default = 80 }
variable "available_ram_mb" {}

locals {
  max_children = floor(var.available_ram_mb / var.worker_ram_mb)
}

# Fail fast if someone sets max_children below a safe floor
resource "null_resource" "validate_pool" {
  triggers = { max_children = local.max_children }
  provisioner "local-exec" {
    command = "[ ${local.max_children} -ge 10 ] || (echo 'pm.max_children too low' && exit 1)"
  }
}

3. Checkov / OPA policy — reject static pools in IaC

# checkov custom check: no static PHP-FPM pools in prod
from checkov.common.models.enums import CheckResult
from checkov.ansible.checks.task.base_ansible_check import BaseAnsibleCheck

class CheckFPMPoolMode(BaseAnsibleCheck):
    def __init__(self):
        super().__init__(
            name="Ensure PHP-FPM pool uses dynamic or ondemand pm",
            id="CKV_ANSIBLE_PHPFPM_1",
            supported_tasks=["ini_file", "template"]
        )
    def get_resource_id(self, conf):
        return conf.get("name", "unknown")
    def scan_resource_conf(self, conf):
        # Flag any config setting pm = static
        if "static" in str(conf):
            return CheckResult.FAILED
        return CheckResult.PASSED

4. Prometheus alerting — fire before users see 502s

# prometheus/alerts/phpfpm.yml
groups:
  - name: phpfpm_pool
    rules:
      - alert: PHPFPMPoolNearExhaustion
        expr: |
          phpfpm_active_processes / phpfpm_max_active_processes > 0.80
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "PHP-FPM pool {{ $labels.pool }} at {{ $value | humanizePercentage }} capacity"
          runbook: "https://wiki.internal/runbooks/phpfpm-exhaustion"

      - alert: PHPFPMPoolExhausted
        expr: |
          phpfpm_active_processes >= phpfpm_max_active_processes
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: "PHP-FPM pool EXHAUSTED — 502s imminent"

5. `request_terminate_timeout` is your last line of defense

Never leave it at 0. A single slow DB query or infinite loop will permanently occupy a worker slot until FPM or the server is restarted. Set it to 1.5× your fastcgi_read_timeout in Nginx so FPM kills the worker before Nginx gives up and logs the error.