PVC Resize Node Restart Validator

The Incident (What Does the Error Mean?)

You ran kubectl get pvc and saw this:

NAME         STATUS   VOLUME           CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-pvc-0   Bound    pvc-a1b2c3d4     20Gi       RWO            gp2            14d

But you patched it to 50Gi. Now:

NAME         STATUS                    VOLUME           CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-pvc-0   Bound                     pvc-a1b2c3d4     20Gi       RWO            gp2            14d

Conditions:
  Type                             Status
  FileSystemResizePending          True   — Waiting for user to (re)start a pod to finish file system resize of volume on node.

Immediate consequence: The block device is already 50Gi at the cloud provider level (EBS, GCE PD, Azure Disk). The pod still sees 20Gi. Any write that would have fit in 50Gi fails. Stateful workloads — Postgres, Kafka, Elasticsearch — will crash with ENOSPC or hang on write.

The Attack Vector / Blast Radius

This is a two-phase resize and Kubernetes only completed phase 1.

Phase 1 — Control plane / cloud API: external-resizer CSI sidecar calls the cloud API (ModifyVolume on EBS, etc.). Block device is now 50Gi. PV object is updated. ✅ Done.

Phase 2 — Node / filesystem: kubelet must detect the mounted volume has a larger block device, then call resize2fs (ext4) or xfs_growfs (XFS) inside the pod's mount namespace. This only happens on pod restart / remount. If the pod is still running, kubelet never gets the trigger.

Blast radius:

• Stateful sets with volumeClaimTemplates will have every replica in this state — one bad rollout and all replicas are capacity-starved simultaneously.
• If allowVolumeExpansion: false on the StorageClass, the PVC patch was silently accepted by the API server but the CSI driver will reject it — leaving the PVC in a permanently inconsistent desired vs. actual state.
• On ReadWriteOnce volumes, the block device is node-locked. A pod rescheduled to a different node will force a detach/reattach, which can take 6–10 minutes on AWS EBS (the multi-attach error window).

Prevention in CI/CD

1. OPA/Gatekeeper — Enforce `allowVolumeExpansion: true` on all StorageClasses

package storageclass

deny[msg] {
  input.kind == "StorageClass"
  not input.allowVolumeExpansion == true
  msg := sprintf("StorageClass '%v' must have allowVolumeExpansion: true", [input.metadata.name])
}

2. Checkov — Block non-expandable StorageClass in Terraform

checkov -d ./terraform --check CKV_K8S_STORAGE_ALLOW_EXPANSION

If using the Kubernetes Terraform provider:

 resource "kubernetes_storage_class" "gp3" {
   metadata { name = "gp3" }
   storage_provisioner = "ebs.csi.aws.com"
+  allow_volume_expansion = true
   parameters = { type = "gp3" }
 }

3. Helm pre-upgrade hook — Validate PVC resize will succeed before rollout

apiVersion: batch/v1
kind: Job
metadata:
  name: pvc-resize-preflight
  annotations:
    "helm.sh/hook": pre-upgrade
    "helm.sh/hook-delete-policy": hook-succeeded
spec:
  template:
    spec:
      containers:
      - name: checker
        image: bitnami/kubectl:latest
        command:
        - /bin/sh
        - -c
        - |
          SC=$(kubectl get pvc $PVC_NAME -o jsonpath='{.spec.storageClassName}')
          EXPAND=$(kubectl get sc $SC -o jsonpath='{.allowVolumeExpansion}')
          [ "$EXPAND" = "true" ] || (echo "StorageClass $SC does not allow expansion. Aborting."; exit 1)

4. AlertManager rule — Alert on `FileSystemResizePending` before it causes ENOSPC

- alert: PVCFilesystemResizePending
  expr: kube_persistentvolumeclaim_status_condition{condition="FileSystemResizePending",status="true"} == 1
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "PVC {{ $labels.persistentvolumeclaim }} filesystem resize pending — pod restart required"