Nginx Worker Connections Exhaustion Fix

The Incident (What Does the Error Mean?)

Raw log output from /var/log/nginx/error.log:

2024/07/15 03:42:17 [alert] 1234#1234: worker_connections are not enough: accept() failed (24: Too many open files)
2024/07/15 03:42:17 [alert] 1234#1234: *8821903 open() "/var/cache/nginx/..." failed (24: Too many open files)

Errno 24 is EMFILE — the process-level open file descriptor table is full. Nginx cannot open a new socket for the incoming connection. The request is silently dropped. No 502, no 503 — the TCP handshake completes at the kernel level but Nginx immediately closes it. Clients see a connection reset. Monitoring that only checks HTTP status codes will miss this entirely.

The Attack Vector / Blast Radius

This is a cascading capacity failure, not a single-point event:

1. Each Nginx connection consumes at minimum 1 fd (client socket). Proxying to an upstream doubles it. Serving a file adds another. A single HTTP/1.1 keepalive session with assets can hold 3–5 fds simultaneously.
2. Default ulimit -n on most Linux distros is 1024. A single Nginx worker with worker_connections 1024 will saturate this under moderate load — the math doesn't work unless you explicitly raise the OS limit.
3. Systemd overrides the shell ulimit. Even if you set ulimit -n 65535 in /etc/security/limits.conf, a systemd-managed Nginx process ignores it unless LimitNOFILE is set in the service unit. This is the #1 reason the fix appears to work in testing and fails in production.
4. Blast radius: 100% of new inbound connections fail. Active keepalive connections already established continue until they close. Traffic spikes (deploys, marketing events, bot crawls) trigger the threshold unpredictably. The worker process does not crash — it keeps running and logging alerts at high frequency, which can itself cause disk I/O pressure.

Prevention in CI/CD

1. Nginx config linting in your pipeline:

# In your CI step — catches syntax errors before deploy
nginx -t -c /path/to/nginx.conf

2. Checkov policy for Nginx fd limits (custom check):

# checkov custom check: ensure worker_rlimit_nofile >= 65535
from checkov.common.models.enums import CheckResult

def check_nginx_rlimit(config_text):
    import re
    match = re.search(r'worker_rlimit_nofile\s+(\d+)', config_text)
    if not match or int(match.group(1)) < 65535:
        return CheckResult.FAILED
    return CheckResult.PASSED

3. Ansible enforcement — ensure the systemd override is always present:

- name: Set Nginx LimitNOFILE via systemd override
  ansible.builtin.copy:
    dest: /etc/systemd/system/nginx.service.d/override.conf
    content: |
      [Service]
      LimitNOFILE=65535
  notify:
    - daemon-reload
    - restart nginx

4. Alerting — add a Prometheus alert on fd exhaustion before it hits 100%:

# prometheus/alerts.yml
- alert: NginxFdExhaustionImminent
  expr: nginx_connections_active / nginx_worker_connections_limit > 0.80
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "Nginx fd pool >80% utilized — preemptive action required"