Docker Cgroups Kernel OOM Monitor

The Incident (What does the error mean?)

Your kernel logs and Docker daemon surface this as:

KERNEL: Memory cgroup out of memory: Kill process 14821 (java) score 1823 or sacrifice child
Killed process 14821 (java) total-vm:4194304kB, anon-rss:2097152kB, file-rss:32768kB
docker: container killed by kernel OOM killer
exited with code 137

Exit code 137 = 128 + 9 (SIGKILL). This is not a graceful shutdown. There is no cleanup, no connection draining, no checkpoint. The process is instantly terminated by the kernel at the moment cgroup memory.usage_in_bytes attempts to exceed memory.limit_in_bytes. Any in-flight transactions, open file handles, or write buffers are gone.

Verify the active cgroup limit on a running container:

# Get the cgroup path
docker inspect <container_id> --format '{{.HostConfig.Memory}}'

# Read directly from cgroup v1
cat /sys/fs/cgroup/memory/docker/<container_id>/memory.limit_in_bytes

# cgroup v2
cat /sys/fs/cgroup/system.slice/docker-<container_id>.scope/memory.max

# Watch OOM events in real time
dmesg -w | grep -i 'oom\|killed'

The Attack Vector / Blast Radius

This is a cascading failure vector, not an isolated container death.

1. Kubernetes restart loops: If this is a K8s pod, OOMKilled status triggers CrashLoopBackOff. The pod restarts with exponential backoff. Under sustained load, it never recovers. Your Deployment is effectively down.
2. Data corruption: Any container writing to a database, message queue, or file without fsync guarantees loses buffered writes. PostgreSQL connections are severed mid-transaction. Kafka producers lose unacknowledged messages.
3. Noisy neighbor amplification: On a shared node, if --memory-swap is not explicitly set, Docker defaults to 2x the memory limit as swap. The container starts thrashing swap before the OOM kill, degrading every other container on the host for minutes before the kill fires.
4. Silent recurrence: Without --restart=on-failure or proper liveness probes, the container stays dead. Monitoring that only checks HTTP endpoints misses this entirely if the port never re-binds.
5. JVM/Node.js specific trap: The JVM does not respect cgroup limits by default below JDK 8u191. It reads /proc/meminfo for total host RAM and sets heap accordingly. A container with --memory=512m on a 64GB host will have the JVM attempt a multi-GB heap. It will be killed every time.

Prevention in CI/CD

The goal is catching under-provisioned memory limits before they hit production.

1. Checkov — static analysis on Compose and K8s manifests:

pip install checkov
checkov -f docker-compose.yml --check CKV_DOCKER_7,CKV_DOCKER_14
checkov -d ./k8s-manifests/ --check CKV_K8S_11,CKV_K8S_13

CKV_K8S_11 flags missing memory limits. CKV_K8S_13 flags missing memory requests.

2. OPA/Gatekeeper — enforce memory limits at admission:

# Deny pods with no memory limit set
violation[{"msg": msg}] {
  container := input.review.object.spec.containers[_]
  not container.resources.limits.memory
  msg := sprintf("Container '%v' has no memory limit. OOM kills will be uncontrolled.", [container.name])
}

3. Load test with memory profiling in staging — not optional:

# Run your load test, then capture peak memory
docker stats --format "{{.MemUsage}}" <container_id> > mem_profile.log &
k6 run --vus 50 --duration 5m load_test.js

Set your --memory limit to peak observed RSS + 25% headroom. Never guess.

4. Alerting — alert on container_memory_working_set_bytes before the kill:

# Prometheus alert rule
- alert: ContainerMemoryNearLimit
  expr: |
    (container_memory_working_set_bytes / container_spec_memory_limit_bytes) > 0.85
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "Container {{ $labels.container }} at {{ $value | humanizePercentage }} of memory limit"

Firing at 85% gives you time to act before the kernel fires at 100%.