Kubernetes Ephemeral Storage Eviction Monitor

The Incident (What Does the Error Mean?)

The raw event from kubectl describe pod <pod-name>:

Status:    Failed
Reason:    Evicted
Message:   The node was low on resource: ephemeral-storage.
           Threshold quantity: 10%, available: 8%.
           Container <container-name> was using 4Gi, request is 0.

And from kubectl get events --field-selector reason=Evicted:

WARNING   Evicted   pod/my-app-7d9f8b-xk2p1   
The node was low on resource: ephemeral-storage. 
Threshold quantity: 10%, available: 8%.

Immediate consequence: The pod is terminated with reason: Evicted and will not be rescheduled onto the same node until the storage pressure clears. If your PodDisruptionBudget is misconfigured or your deployment has replicas: 1, this is a full service outage. The evicted pod is not automatically restarted in-place — the ReplicaSet controller creates a new pod, which may land on the same saturated node and get evicted again in a loop.

The Attack Vector / Blast Radius

Ephemeral storage is consumed from three sources the kubelet tracks:

1. Container overlay (writable layer): Every docker/containerd write that isn't to a mounted volume hits the node's local disk.
2. Container logs: stdout/stderr from your container are written to /var/log/pods/ on the node. A chatty logger with no log rotation will silently eat gigabytes.
3. emptyDir volumes: Scratch space mounted into the pod. Unless sizeLimit is set, this is unbounded and counts toward the pod's ephemeral storage usage.

Cascading failure scenario:

• Node disk fills → kubelet triggers soft eviction (warning) → if not resolved, hard eviction fires → all pods on that node without guaranteed QoS for ephemeral storage are candidates for eviction, ordered by usage-over-request ratio.
• A single log-spewing pod can evict unrelated, well-behaved pods on the same node. This is a noisy neighbor problem at the infrastructure layer.
• If the node is part of a small cluster (2–3 nodes), cascading evictions across nodes under load can collapse your entire deployment.
• DaemonSets are not immune. A DaemonSet pod evicted for storage pressure will be recreated immediately — and evicted again — creating a tight eviction loop that hammers the kubelet.

Prevention in CI/CD

1. Block zero-limit deployments with OPA/Gatekeeper

Deploy a ConstraintTemplate that rejects any pod spec missing ephemeral-storage limits:

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: requireephemeralstoragelimits
spec:
  crd:
    spec:
      names:
        kind: RequireEphemeralStorageLimits
  targets:
  - target: admission.k8s.gatekeeper.sh
    rego: |
      package requireephemeralstoragelimits
      violation[{"msg": msg}] {
        container := input.review.object.spec.containers[_]
        not container.resources.limits["ephemeral-storage"]
        msg := sprintf("Container '%v' must have ephemeral-storage limits set.", [container.name])
      }

2. Shift-left with Checkov in your pipeline

# .github/workflows/k8s-lint.yaml
- name: Checkov Kubernetes Scan
  uses: bridgecrewio/checkov-action@master
  with:
    directory: ./k8s/
    check: CKV_K8S_20,CKV_K8S_11  # resource limits + requests checks
    framework: kubernetes
    soft_fail: false

CKV_K8S_11 specifically checks for memory/CPU limits. Add a custom Checkov policy for ephemeral storage:

# checkov/custom_checks/ephemeral_storage_limit.py
from checkov.common.models.enums import CheckResult, CheckCategories
from checkov.kubernetes.checks.resource.base_container_check import BaseK8Check

class EphemeralStorageLimitCheck(BaseK8Check):
    def __init__(self):
        name = "Ensure ephemeral-storage limit is set"
        id = "CKV2_K8S_CUSTOM_1"
        categories = [CheckCategories.GENERAL_SECURITY]
        super().__init__(name=name, id=id, categories=categories,
                         supported_entities=['containers'])

    def scan_resource_conf(self, conf):
        limits = conf.get("resources", {}).get("limits", {})
        if limits.get("ephemeral-storage"):
            return CheckResult.PASSED
        return CheckResult.FAILED

scanner = EphemeralStorageLimitCheck()

3. Monitor with Prometheus alerts before eviction fires

# prometheus-rules.yaml
groups:
- name: ephemeral-storage
  rules:
  - alert: NodeEphemeralStoragePressureWarning
    expr: |
      kubelet_node_name * on(node) group_left()
      (1 - (node_filesystem_avail_bytes{mountpoint="/"} 
            / node_filesystem_size_bytes{mountpoint="/"})) > 0.75
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Node {{ $labels.node }} ephemeral storage > 75% full"
  - alert: PodEphemeralStorageNearLimit
    expr: |
      (kubelet_container_log_filesystem_used_bytes 
       / on(pod, container, namespace) 
       kube_pod_container_resource_limits{resource="ephemeral-storage"}) > 0.8
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "Container {{ $labels.container }} in {{ $labels.namespace }} at 80% ephemeral-storage limit"

This alert fires before the kubelet eviction threshold is hit, giving your on-call engineer time to act rather than react.