Multi-AZ EBS VolumeAttachment State Checker

The Incident (What Does the Error Mean?)

Raw state from kubectl describe volumeattachment <name>:

Name:         csi-abc123def456
Namespace:
Labels:       <none>
Status:
  Attached:   false
Attach Error:
  Message:    context deadline exceeded
Spec:
  Attacher:   ebs.csi.aws.com
  Node Name:  ip-10-0-2-45.ec2.internal
  Source:
    Persistent Volume Name: pvc-7f3e1a2b

And from kubectl get events -n <namespace>:

Warning  FailedAttachVolume  pod/my-statefulset-0  
Multi-Attach error for volume "pvc-7f3e1a2b": 
Volume is already exclusively attached to one node 
and can't be attached to another

Immediate consequence: The pod is stuck in ContainerCreating. The previous node (possibly terminated or drained) still holds the VolumeAttachment object in the API server. AWS EBS io1/gp3 volumes are ReadWriteOnce — they cannot attach to two nodes simultaneously, even across AZs. Kubernetes is waiting for the old attachment to release. It won't, because the node is gone.

The Attack Vector / Blast Radius

This is a StatefulSet availability bomb in disguise. Here's the cascade:

1. Node failure or spot interruption in us-east-1a — the node object is deleted but the VolumeAttachment object persists in etcd.
2. Kubernetes reschedules the pod to a node in us-east-1b. The EBS CSI driver attempts to attach the volume. AWS rejects it — the volume is AZ-locked AND the old attachment record exists.
3. The attach-detach controller has a default 6-minute timeout before it force-detaches. In production, this means 6+ minutes of guaranteed downtime per affected pod.
4. If volumeBindingMode: Immediate is set on the StorageClass, the PVC was bound to an AZ at provision time with zero awareness of where pods will actually land. Every rescheduling event across AZs is a guaranteed attach failure.
5. For StatefulSets with multiple replicas, each replica hitting this on the same node failure event multiplies the outage window.

Secondary blast: if your node-problem-detector or cluster autoscaler is aggressively terminating nodes, you can enter a loop where volumes are perpetually in attaching state across a rolling set of nodes.

Prevention in CI/CD

1. Conftest / OPA Policy — Block `Immediate` Binding Mode

package kubernetes.storage

deny[msg] {
  input.kind == "StorageClass"
  input.provisioner == "ebs.csi.aws.com"
  input.volumeBindingMode != "WaitForFirstConsumer"
  msg := sprintf(
    "StorageClass '%v' must use WaitForFirstConsumer for EBS in multi-AZ clusters",
    [input.metadata.name]
  )
}

Run in CI:

conftest test storageclass.yaml --policy policy/

2. Checkov — Terraform EBS Validation

checkov -d ./terraform --check CKV_AWS_189,CKV_AWS_3
# CKV_AWS_189: EBS encryption enabled
# CKV_AWS_3: EBS volume not publicly restorable

Add a custom Checkov check for WaitForFirstConsumer if managing StorageClass via Helm values in Terraform.

3. Alerting — Don't Rely on Manual Discovery

Add this Prometheus alert. Six minutes of silent attaching is too long to find manually:

- alert: EBSVolumeAttachmentStuck
  expr: |
    kube_volumeattachment_info{attacher="ebs.csi.aws.com"} == 1
    and on(volumeattachment)
    kube_volumeattachment_status_attached == 0
  for: 3m
  labels:
    severity: critical
  annotations:
    summary: "EBS VolumeAttachment stuck for >3m: {{ $labels.volumeattachment }}"
    runbook_url: "https://your-runbook/ebs-stuck-attachment"

4. Node Termination Handler

Deploy the AWS Node Termination Handler for spot instances. It cordons and drains nodes gracefully before termination, giving the attach-detach controller time to cleanly release VolumeAttachment objects before the node disappears.

helm install aws-node-termination-handler \
  eks/aws-node-termination-handler \
  --namespace kube-system \
  --set enableSpotInterruptionDraining=true \
  --set enableRebalanceMonitoring=true