Docker Ulimit Open Files Optimizer

The Incident (What Does the Error Mean?)

Raw error output from container logs or host dmesg:

Error response from daemon: OCI runtime create failed:
container_linux.go:380: starting container process caused:
process_linux.go:545: container init caused:
rootfs_linux.go:76: mounting "/dev/null" caused:
open /dev/null: too many open files

or in application logs:

accept tcp 0.0.0.0:8080: accept: too many open files
ERROR: EMFILE: too many open files, open '/app/config.json'

Immediate consequence: The process cannot open any new file descriptors — no new TCP connections, no log file writes, no IPC pipes. The container is effectively dead to new traffic while existing goroutines/threads spin or panic.

The Attack Vector / Blast Radius

This is a resource exhaustion failure, not a code bug. The Linux kernel enforces a per-process file descriptor limit via ulimit -n. Docker inherits the host daemon's default (nofile: 1024) and passes it into every container unless explicitly overridden.

Cascading failure chain:

1. High-concurrency service (Node.js, Nginx, Java NIO, Go net/http) opens one FD per connection + one per log file + one per DB pool connection.
2. FD count hits 1024. All accept() syscalls return EMFILE.
3. Load balancer health checks fail. Container is marked unhealthy.
4. Orchestrator (ECS, Kubernetes, Swarm) restarts the container.
5. On restart, in-flight requests are dropped. If the underlying cause (traffic spike, FD leak) isn't fixed, the new container hits the same ceiling within seconds — restart loop.
6. In a microservices mesh, the failing container backs up upstream callers, triggering timeout storms across dependent services.

FD leak multiplier: If your app has a file descriptor leak (unclosed sockets, missing defer f.Close()), a low nofile limit accelerates the failure timeline from hours to minutes under load.

Prevention in CI/CD

1. Checkov Policy (IaC Scanning)

Add to your pipeline to catch missing ulimits in Compose files before merge:

# .checkov.yml
checks:
  - CKV_DOCKER_ULIMIT_NOFILE  # custom check or use checkov --check CKV2_DOCKER_*

Write a custom Checkov check:

# checkov/custom/check_ulimit_nofile.py
from checkov.common.models.enums import CheckResult
from checkov.dockerfile.checks.base_dockerfile_check import BaseDockerfileCheck

class UlimitNofileCheck(BaseDockerfileCheck):
    def __init__(self):
        super().__init__(
            name="Ensure ulimit nofile >= 65536",
            id="CKV_CUSTOM_DOCKER_1",
            supported_instructions=["RUN"]
        )

2. OPA/Conftest Policy for `docker-compose.yml`

# policy/ulimit.rego
package compose

deny[msg] {
  service := input.services[name]
  not service.ulimits.nofile.hard
  msg := sprintf("Service '%v' missing ulimits.nofile.hard", [name])
}

deny[msg] {
  service := input.services[name]
  service.ulimits.nofile.hard < 65536
  msg := sprintf("Service '%v' ulimits.nofile.hard must be >= 65536", [name])
}

Run in CI:

conftest test docker-compose.yml --policy policy/

3. Host-Level Baseline (Ansible/Terraform)

Enforce /etc/docker/daemon.json via configuration management so no host is provisioned without the correct default:

# terraform/modules/ec2_docker_host/userdata.sh
cat > /etc/docker/daemon.json <<EOF
{
  "default-ulimits": {
    "nofile": { "Name": "nofile", "Hard": 65536, "Soft": 65536 }
  }
}
EOF
systemctl reload docker

4. Monitor FD Usage Before You Hit the Wall

# Per-container FD count (run on host)
for pid in $(docker inspect --format '{{.State.Pid}}' $(docker ps -q)); do
  echo "PID $pid: $(ls /proc/$pid/fd 2>/dev/null | wc -l) FDs"
done

Alert when any container exceeds 80% of its nofile hard limit — not after it crashes.