StatefulSet Scaling Volume Mount Verifier

Threat/Impact Level: HIGH | Downtime Risk: HIGH | Time to Fix: 15–45 mins

TL;DR

• What broke: After scaling a StatefulSet to 0 and back up, the pod is stuck in ContainerCreating because its PVC is orphaned, the VolumeAttachment object is stale, or the volume is still attached to the previous node.
• How to fix it: Force-delete the stale VolumeAttachment, verify the PVC is Bound, and ensure your volumeClaimTemplates name exactly matches the volumeMounts name in the pod spec.
• Shortcut: Use our Client-Side Sandbox above to auto-refactor your StatefulSet YAML and surface the exact field mismatch.

Warning  FailedMount  4m    kubelet  MountVolume.SetUp failed for volume "data" :
         rpc error: code = Internal desc = volume data is not mounted
Warning  FailedMount  2m    kubelet  Unable to attach or mount volumes: unmounted volumes=[data],
         unattached volumes=[data]: timed out waiting for the condition

FailedAttachVolume  Multi-Attach error for volume "pvc-xxxxxxxx" :
  Volume is already exclusively attached to one node and can't be attached to another

How to Fix It

Step 1: Diagnose the Actual State

# Check PVC status — must be Bound, not Released or Pending
kubectl get pvc -n <namespace> -l app=<statefulset-name>

# Check for stale VolumeAttachment objects
kubectl get volumeattachment | grep <pv-name>

# Get the PV name from the PVC
kubectl get pvc <pvc-name> -n <namespace> -o jsonpath='{.spec.volumeName}'

# Describe the pod for exact mount error
kubectl describe pod <pod-name> -n <namespace> | grep -A 20 "Events"

Basic Fix: Delete the Stale VolumeAttachment

# Identify the stale attachment
kubectl get volumeattachment

# Force delete it — the CSI driver will re-create it correctly on next attach
kubectl delete volumeattachment <attachment-name>

# If stuck in Terminating, patch out the finalizer
kubectl patch volumeattachment <attachment-name> \
  -p '{"metadata":{"finalizers":null}}' --type=merge

After deletion, the pod's next attach cycle will succeed within 60–90 seconds.

Enterprise Best Practice: Fix the Root Config

The most common permanent cause is a name mismatch between volumeClaimTemplates and volumeMounts, or a missing storageClassName causing WaitForFirstConsumer to deadlock.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: "postgres"
  replicas: 3
  podManagementPolicy: OrderedReady
  template:
    spec:
      containers:
      - name: postgres
        image: postgres:15
        volumeMounts:
-         - name: data-volume        # WRONG: does not match volumeClaimTemplate name
+         - name: data               # CORRECT: must exactly match .volumeClaimTemplates[].metadata.name
            mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
  - metadata:
-     name: data-volume             # WRONG: mismatch causes silent mount failure
+     name: data                    # CORRECT
    spec:
-     storageClassName: ""          # WRONG: empty string disables dynamic provisioning
+     storageClassName: "gp3-csi"   # CORRECT: explicit class with WaitForFirstConsumer
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
-         storage: 1Gi              # WRONG: undersized, causes resize churn
+         storage: 20Gi

For EBS/GCE PD — prevent Multi-Attach errors with node affinity enforcement:

# StorageClass definition
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3-csi
provisioner: ebs.csi.aws.com
- volumeBindingMode: Immediate        # WRONG: binds before pod is scheduled, causes node mismatch
+ volumeBindingMode: WaitForFirstConsumer  # CORRECT: binds to the node the pod lands on
  reclaimPolicy: Retain
  parameters:
    type: gp3
+   encrypted: "true"

💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.

(For CI/CD pipeline automation, install via npm i -g stackengine).

# opa/statefulset-volume-name-match.rego
package kubernetes.admission

deny[msg] {
  input.request.kind.kind == "StatefulSet"
  container := input.request.object.spec.template.spec.containers[_]
  mount := container.volumeMounts[_]
  claim_names := {t.metadata.name | t := input.request.object.spec.volumeClaimTemplates[_]}
  not claim_names[mount.name]
  msg := sprintf("volumeMount '%v' has no matching volumeClaimTemplate", [mount.name])
}

# .github/workflows/k8s-lint.yml
- name: Checkov StatefulSet Scan
  uses: bridgecrewio/checkov-action@master
  with:
    directory: ./k8s/
    check: CKV_K8S_28,CKV_K8S_6
    framework: kubernetes
    soft_fail: false

#!/bin/bash
# pre-scale-down.sh — run before kubectl scale --replicas=0
PV_NAME=$(kubectl get pvc data-postgres-0 -o jsonpath='{.spec.volumeName}')
ATTACHMENT=$(kubectl get volumeattachment -o json | \
  jq -r ".items[] | select(.spec.source.persistentVolumeName==\"$PV_NAME\") | .metadata.name")

if [ -n "$ATTACHMENT" ]; then
  echo "WARNING: VolumeAttachment $ATTACHMENT exists. Deleting before scale-down."
  kubectl delete volumeattachment "$ATTACHMENT"
  sleep 10
fi

kubectl scale statefulset postgres --replicas=0

# Alert fires if any pod is stuck in ContainerCreating > 5 minutes
- alert: StatefulSetVolumeMountStuck
  expr: |
    kube_pod_container_status_waiting_reason{reason="ContainerCreating"} == 1
    and on(pod) kube_pod_owner{owner_kind="StatefulSet"}
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "StatefulSet pod {{ $labels.pod }} stuck mounting volume"
    runbook: "https://your-wiki/runbooks/statefulset-volume-mount"