K8s DiskPressure Container Log Purger

Threat/Impact Level: HIGH | Downtime Risk: HIGH | Time to Fix: 15–30 mins

TL;DR

• What broke: Kubernetes worker nodes crossed the imagefs.available or nodefs.available eviction threshold because container log files under /var/log/pods/ or /var/lib/docker/containers/ were never rotated, consuming 80%+ of node disk.
• How to fix it: Enforce containerLogMaxSize and containerLogMaxFiles in the kubelet config, and immediately purge orphaned log files with find + truncate.
• Fast path: Use our Client-Side Sandbox above to auto-refactor your kubelet ConfigMap or container spec — paste your config and get corrected YAML instantly.

Conditions:
  Type              Status
  ----              ------
  DiskPressure      True

Events:
  Warning  Evicted   pod/api-server-7d9f4b8c6-xk2p9   
  The node had condition: DiskPressure

WARNING  Evicting pod api-server-7d9f4b8c6-xk2p9 
because it exceeded its local ephemeral storage limit 
or the node is under DiskPressure.

How to Fix It

Step 0: Immediate Triage (Run This First)

# Identify top disk consumers on the affected node (run via node shell or SSH)
du -sh /var/log/pods/* | sort -rh | head -20
du -sh /var/lib/containerd/* | sort -rh | head -10

# Truncate (do NOT delete — deletion of open files won't free inodes) active log files over 500MB
find /var/log/pods/ -name '*.log' -size +500M -exec truncate -s 0 {} \;

# Force kubelet to GC dead containers
kubectl delete pod --field-selector=status.phase=Failed -A

Basic Fix: Kubelet Log Rotation Config

Edit the kubelet config on each affected node. Location is typically /etc/kubernetes/kubelet-config.yaml or managed via a ConfigMap in kubeadm clusters.

 apiVersion: kubelet.config.k8s.io/v1beta1
 kind: KubeletConfiguration
-# containerLogMaxSize and containerLogMaxFiles not set (defaults: 10Mi / 5)
+containerLogMaxSize: "50Mi"
+containerLogMaxFiles: 3
 evictionHard:
-  nodefs.available: "5%"
+  nodefs.available: "10%"
+  nodefs.inodesFree: "5%"
+imageGCHighThresholdPercent: 75
+imageGCLowThresholdPercent: 70

After editing, restart kubelet:

systemctl daemon-reload && systemctl restart kubelet

Enterprise Best Practice: Enforce via DaemonSet + Logrotate + Resource Quotas

1. Container-level log limits in the Pod spec (defense-in-depth):

 containers:
 - name: api-server
   image: myapp:v2.1.0
+  resources:
+    limits:
+      ephemeral-storage: "2Gi"
+    requests:
+      ephemeral-storage: "512Mi"

2. Cluster-level enforcement via LimitRange:

+apiVersion: v1
+kind: LimitRange
+metadata:
+  name: ephemeral-storage-limits
+  namespace: production
+spec:
+  limits:
+  - type: Container
+    max:
+      ephemeral-storage: "4Gi"
+    default:
+      ephemeral-storage: "1Gi"
+    defaultRequest:
+      ephemeral-storage: "256Mi"

3. Node-level logrotate DaemonSet for legacy containerd/docker log paths:

+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: log-purger
+  namespace: kube-system
+spec:
+  selector:
+    matchLabels:
+      app: log-purger
+  template:
+    spec:
+      tolerations:
+      - operator: Exists  # Run even on tainted/pressured nodes
+      hostPID: true
+      containers:
+      - name: log-purger
+        image: alpine:3.19
+        command:
+        - /bin/sh
+        - -c
+        - |
+          while true; do
+            find /host/var/log/pods -name '*.log' -size +200M \
+              -exec truncate -s 0 {} \;
+            find /host/var/log/pods -maxdepth 3 -type d -empty -delete
+            sleep 300
+          done
+        volumeMounts:
+        - name: varlog
+          mountPath: /host/var/log
+      volumes:
+      - name: varlog
+        hostPath:
+          path: /var/log

💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.

(For CI/CD pipeline automation, install via npm i -g stackengine).

package kubernetes.admission

deny[msg] {
  input.request.kind.kind == "Pod"
  container := input.request.object.spec.containers[_]
  not container.resources.limits["ephemeral-storage"]
  msg := sprintf("Container '%v' must define ephemeral-storage limit", [container.name])
}

# Fails pipeline if ephemeral-storage limits are missing
checkov -d ./k8s-manifests \
  --check CKV_K8S_20 \
  --check CKV_K8S_11

- alert: NodeDiskPressureImminent
  expr: |
    (node_filesystem_avail_bytes{mountpoint="/"} /
     node_filesystem_size_bytes{mountpoint="/"}) < 0.15
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Node {{ $labels.instance }} disk below 15% — eviction imminent"

kube-bench node --check 4.2.10,4.2.11
# Validates containerLogMaxSize and containerLogMaxFiles are set