Docker Overlay2 Inode Exhaustion Monitor

The Incident (What Does the Error Mean?)

Raw error surface — you'll see this in dockerd logs, journalctl -u docker, or directly in your CI runner output:

error creating overlay mount to /var/lib/docker/overlay2/b3f9.../merged: 
  too many open files in system

Error response from daemon: 
  failed to create rootfs: 
  mkdir /var/lib/docker/overlay2/c7a1.../ : 
  no space left on device

[containerd] failed to reserve snapshot: 
  write /var/lib/docker/overlay2/...: 
  no space left on device

The trap: df -h returns 40% free. Engineers waste 20 minutes looking at disk blocks. The real culprit is inodes:

$ df -i /var/lib/docker
Filesystem      Inodes   IUsed  IFree IUse% Mounted on
/dev/xvda1     1310720 1310720      0  100% /

IFree: 0. Every overlay2 layer directory, every file inside a container layer, every whiteout file consumes one inode. On a default ext4 partition formatted with mkfs.ext4 defaults (~1 inode per 16KB), a host running hundreds of containers with thousands of small files will hit inode exhaustion long before block exhaustion.

Immediate consequence: No new containers start. Running containers that need to write new files to their writable layer may also begin failing. docker build aborts mid-layer. Kubernetes nodes report NodeNotReady if the container runtime cannot create pod sandboxes.

The Attack Vector / Blast Radius

This is a cascading infrastructure failure, not an isolated Docker issue.

Blast radius chain:

1. dockerd cannot mkdir new overlay2 merge directories → all docker run calls return non-zero exit codes.
2. Kubernetes kubelet CRI calls to containerd fail → pods stuck in ContainerCreating indefinitely.
3. Liveness/readiness probes on existing pods begin failing if those containers attempt filesystem writes (logging, temp files, pid files).
4. CI/CD pipelines (GitHub Actions self-hosted, GitLab Runner, Jenkins agents) fail every job — your entire deployment pipeline is down.
5. Log rotation daemons (logrotate, fluentd) fail to create new log files → logs lost or processes crash.
6. If this host is a Kubernetes node, the node taints itself node.kubernetes.io/disk-pressure and the scheduler stops placing new pods on it — but existing pods may also be evicted depending on eviction thresholds.

Why it's insidious: Monitoring alerts on disk block usage (the standard) will show green. Only an explicit df -i check or an inode-specific alert catches this. Most teams don't have it.

Prevention in CI/CD

1. Add Inode Monitoring to Your Alerting Stack

Prometheus + node_exporter (this is the one alert most teams are missing):

# alerts/docker-inodes.yaml
groups:
  - name: inode_exhaustion
    rules:
      - alert: FilesystemInodesExhaustionWarning
        expr: node_filesystem_files_free{mountpoint="/var/lib/docker"} / node_filesystem_files{mountpoint="/var/lib/docker"} < 0.15
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Docker overlay2 inode usage above 85% on {{ $labels.instance }}"

      - alert: FilesystemInodesExhaustionCritical
        expr: node_filesystem_files_free{mountpoint="/var/lib/docker"} / node_filesystem_files{mountpoint="/var/lib/docker"} < 0.05
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Docker overlay2 inode exhaustion imminent on {{ $labels.instance }}"

2. Scheduled Prune in CI/CD Runners

For self-hosted GitHub Actions runners or GitLab Runner hosts, add a cron job:

# /etc/cron.d/docker-prune
0 2 * * * root docker system prune -f --filter "until=24h" >> /var/log/docker-prune.log 2>&1

3. Terraform / Packer: Enforce XFS at Provisioning Time

-data_device_type = "ext4"
+data_device_type = "xfs"
 # In your Packer AMI builder or Terraform aws_instance user_data,
 # always format the Docker data volume as XFS.

4. Checkov / OPA Policy (Kubernetes)

Enforce that DaemonSets running on Docker nodes include inode monitoring sidecars or that node pools use XFS-formatted instance types via a custom OPA Gatekeeper constraint:

# opa/policies/docker_node_filesystem.rego
package kubernetes.admission

deny[msg] {
  input.request.kind.kind == "NodePool"
  not input.request.object.spec.config.diskType == "pd-ssd-xfs"
  msg := "NodePool must use XFS-formatted SSD to prevent overlay2 inode exhaustion"
}

5. Dockerfile Best Practices to Reduce Inode Pressure

-RUN apt-get update
-RUN apt-get install -y curl
-RUN apt-get clean
+# Collapse RUN commands — each RUN creates a new layer with its own inode tree
+RUN apt-get update && \
+    apt-get install -y curl && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*

Each RUN instruction in a Dockerfile creates a separate overlay2 layer directory. Collapsing them reduces the total inode footprint per image by 60–80% in package-heavy images.