ContainerCreating Volume Attach Timeout Auditor

The Incident (What Does the Error Mean?)

Raw event output from kubectl describe pod <pod-name>:

Warning  FailedMount  3m    kubelet  Unable to attach or mount volumes: 
  unmounted volumes=[data-vol], unattached volumes=[data-vol kube-api-access-xxxxx]: 
  timed out waiting for the condition

Warning  FailedAttachVolume  3m  attachdetach-controller  
  AttachVolume.Attach failed for volume "pvc-a1b2c3d4" : 
  context deadline exceeded

Immediate consequence: The pod never exits ContainerCreating. All containers inside are blocked. If this is a Deployment, the rollout stalls. If it's a StatefulSet, the entire ordinal sequence halts — every subsequent pod waits behind this one.

The Attack Vector / Blast Radius

This is not a soft degradation. It is a hard availability failure with the following cascade:

1. StatefulSet deadlock: Pod-0 stuck → Pod-1 never scheduled → quorum-dependent workloads (Kafka, etcd, Postgres replicas) lose a member and may lose write quorum entirely.
2. Node drain amplification: If the volume was previously attached to a node that was drained or terminated without a clean detach, the cloud provider's block device (EBS, GCE PD, Azure Disk) is still marked in-use. The new node cannot attach it. The VolumeAttachment object in the Kubernetes API is orphaned and will not self-heal without intervention.
3. AZ mismatch silent killer: A ReadWriteOnce PV provisioned in us-east-1a cannot attach to a node rescheduled into us-east-1b. The scheduler does not enforce this by default unless volumeBindingMode: WaitForFirstConsumer is set on the StorageClass. This is the single most common root cause in auto-scaling clusters.
4. CSI driver pod eviction: If the csi-node DaemonSet pod was evicted or OOMKilled on the target node, every mount request on that node silently times out. No obvious error surfaces until you check the CSI pod logs directly.

Prevention in CI/CD

1. Enforce `WaitForFirstConsumer` with OPA/Gatekeeper

package k8sstorage

deny[msg] {
  input.review.object.kind == "StorageClass"
  input.review.object.volumeBindingMode != "WaitForFirstConsumer"
  msg := sprintf(
    "StorageClass '%v' must use volumeBindingMode: WaitForFirstConsumer to prevent AZ attach failures.",
    [input.review.object.metadata.name]
  )
}

Deploy as a ConstraintTemplate + K8sStorageBindingMode constraint. This blocks any StorageClass using Immediate from being applied to the cluster.

2. Checkov static scan in your Terraform pipeline

If you provision StorageClasses or EBS volumes via Terraform:

checkov -d ./terraform --check CKV_K8S_28

CKV_K8S_28 flags storage configurations that risk data exposure or misconfiguration. Add custom checks for binding mode via Checkov's Python SDK.

3. Alert on stuck ContainerCreating in your monitoring stack

# Prometheus alerting rule
- alert: PodStuckContainerCreating
  expr: |
    kube_pod_container_status_waiting_reason{reason="ContainerCreating"} == 1
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Pod {{ $labels.pod }} stuck in ContainerCreating for >5m"
    runbook_url: "https://your-wiki/runbooks/volume-attach-timeout"

4. Node termination lifecycle hook (AWS)

For EKS with Cluster Autoscaler or Karpenter, ensure nodes drain cleanly before termination to guarantee volume detach:

 apiVersion: karpenter.sh/v1alpha5
 kind: Provisioner
 spec:
+  ttlSecondsUntilExpired: 2592000
+  ttlSecondsAfterEmpty: 30
   provider:
     instanceProfile: KarpenterNodeInstanceProfile

Pair this with the AWS Node Termination Handler DaemonSet to intercept Spot interruption and ASG lifecycle events, cordon/drain the node, and allow the attach-detach controller to cleanly release volumes before the instance disappears.