Kubernetes Volume Multi-Attach Lock Fix

The Incident (What Does the Error Mean?)

Raw event from kubectl describe pod <pod-name>:

Warning  FailedAttachVolume  3m    attachdetach-controller
Multi-Attach error for volume "pvc-a1b2c3d4-e5f6-7890-abcd-ef1234567890"
Volume is already exclusively attached to one node and can't be attached to another

Also visible in kubectl get events -n <namespace>:

FailedMount  Unable to attach or mount volumes: unmounted volumes=[data],
unattached volumes=[data]: timed out waiting for the condition

Immediate consequence: The pod is stuck in ContainerCreating. The underlying EBS volume (AWS), GCE PD, or Azure Disk — all ReadWriteOnce block devices — has an in-use lock registered in the Kubernetes VolumeAttachment API object pointing at the old node. The new node cannot acquire the lock. Your workload is down.

The Attack Vector / Blast Radius

This is a cascading node failure scenario, not a misconfiguration you can ignore until the next sprint.

1. Node goes NotReady (spot interruption, OOM kill, kernel panic, network partition).
2. Kubernetes waits for the default node.kubernetes.io/unreachable taint toleration timeout — 300 seconds by default — before evicting pods.
3. The cloud provider's volume detach call either never fires or hangs because the node's kubelet is dead and can't unmount cleanly.
4. The attachdetach-controller sees the VolumeAttachment object still exists and refuses to create a new one for the target node. It will not force-detach by default.
5. Every pod in the Deployment/StatefulSet that needs this PVC is now unschedulable. If this is a StatefulSet with podManagementPolicy: OrderedReady, the entire StatefulSet rolls to a halt — not just one pod.
6. StatefulSets are the worst case. Because StatefulSets bind PVCs to specific pod identities, you cannot simply delete the PVC and recreate it. The volume lock must be cleared surgically.

Prevention in CI/CD

1. OPA/Gatekeeper: Block `ReadWriteOnce` on multi-replica Deployments

package kubernetes.admission

deny[msg] {
  input.request.kind.kind == "Deployment"
  replicas := input.request.object.spec.replicas
  replicas > 1
  volume := input.request.object.spec.template.spec.volumes[_]
  pvc := volume.persistentVolumeClaim
  # Flag for manual review — teams must explicitly use RWX for multi-replica
  msg := sprintf("Deployment '%v' has %v replicas with a PVC. Verify PVC accessMode is ReadWriteMany.", [input.request.object.metadata.name, replicas])
}

2. Checkov: Scan StorageClass for `Immediate` binding mode

# .checkov.yaml
checks:
  - CKV2_K8S_STORAGECLASS_BINDING  # flags volumeBindingMode: Immediate

If Checkov doesn't have this built-in for your version, add a custom check:

from checkov.common.models.enums import CheckResult
from checkov.kubernetes.checks.resource.base_resource_check import BaseK8Check

class StorageClassBindingMode(BaseK8Check):
    def __init__(self):
        super().__init__(name="Ensure StorageClass uses WaitForFirstConsumer",
                         check_id="CKV_K8S_CUSTOM_SC_BINDING",
                         supported_entities=['StorageClass'])
    def scan_resource_conf(self, conf):
        if conf.get('volumeBindingMode') == 'WaitForFirstConsumer':
            return CheckResult.PASSED
        return CheckResult.FAILED

3. Terraform: Enforce binding mode at provisioning time

resource "kubernetes_storage_class" "fast" {
  metadata { name = "fast-ssd" }
  storage_provisioner = "ebs.csi.aws.com"
- volume_binding_mode = "Immediate"
+ volume_binding_mode = "WaitForFirstConsumer"
  reclaim_policy      = "Retain"
}

4. Alert before the outage: PagerDuty/Alertmanager rule

- alert: VolumeAttachmentStuck
  expr: kube_volumeattachment_info{attached="true"} * on(node) (kube_node_status_condition{condition="Ready",status="true"} == 0) > 0
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "VolumeAttachment stuck on NotReady node {{ $labels.node }}"
    runbook_url: "https://your-wiki/runbooks/volume-multi-attach"

This fires before the pod reschedule fails, giving you a 2-minute window to intervene or let automation handle the VolumeAttachment deletion.