EBS CSI External Provisioner PVC Debugger

Threat/Impact Level: HIGH | Downtime Risk: HIGH | Time to Fix: 10–30 mins depending on root cause

TL;DR

• What broke: Your PVC is stuck in Pending because the EBS CSI external provisioner cannot fulfill the volume request — caused by a missing/misconfigured CSI driver, wrong provisioner field in the StorageClass, IAM role missing ec2:CreateVolume permissions, or an AZ topology mismatch.
• How to fix it: Verify the ebs.csi.aws.com driver is running, confirm IRSA/node IAM permissions, and ensure your StorageClass references the correct provisioner and AZ topology constraints.
• Shortcut: Use our Client-Side Sandbox above to auto-refactor this — paste your kubectl describe pvc output and StorageClass YAML and get a corrected manifest instantly.

Events:
  Type     Reason                Age                From                         Message
  ----     ------                ----               ----                         -------
  Warning  ProvisioningFailed    2m (x12 over 10m)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created

How to Fix It (The Solution)

Step 0 — Triage in 60 Seconds

# Is the CSI driver running?
kubectl get pods -n kube-system -l app=ebs-csi-controller

# What is the provisioner on your StorageClass?
kubectl get sc <your-storage-class> -o jsonpath='{.provisioner}'

# Full event log on the stuck PVC
kubectl describe pvc <pvc-name> -n <namespace>

# CSI controller logs
kubectl logs -n kube-system -l app=ebs-csi-controller -c csi-provisioner --tail=50

Fix 1 — Wrong Provisioner in StorageClass (Most Common)

 apiVersion: storage.k8s.io/v1
 kind: StorageClass
 metadata:
   name: gp3-ebs
- provisioner: kubernetes.io/aws-ebs
+ provisioner: ebs.csi.aws.com
 parameters:
-  type: gp2
+  type: gp3
+  encrypted: "true"
 reclaimPolicy: Retain
 volumeBindingMode: WaitForFirstConsumer
 allowVolumeExpansion: true

Critical: volumeBindingMode: WaitForFirstConsumer is mandatory for EBS. Without it, the provisioner tries to create the volume before knowing which AZ the pod will schedule into, causing topology failures.

Fix 2 — Missing IAM Permissions (IRSA)

The CSI controller pod must assume a role with at minimum:

 # IAM Policy attached to the CSI controller IRSA role
 {
   "Version": "2012-10-17",
   "Statement": [
     {
       "Effect": "Allow",
       "Action": [
-        "ec2:*"
+        "ec2:CreateVolume",
+        "ec2:DeleteVolume",
+        "ec2:AttachVolume",
+        "ec2:DetachVolume",
+        "ec2:DescribeVolumes",
+        "ec2:DescribeVolumeStatus",
+        "ec2:DescribeInstances",
+        "ec2:DescribeAvailabilityZones",
+        "ec2:CreateTags",
+        "ec2:ModifyVolume",
+        "ec2:DescribeVolumesModifications"
       ],
       "Resource": "*"
     }
   ]
 }

Annotate the CSI service account:

kubectl annotate serviceaccount ebs-csi-controller-sa \
  -n kube-system \
  eks.amazonaws.com/role-arn=arn:aws:iam::<ACCOUNT_ID>:role/AmazonEKS_EBS_CSI_DriverRole

Then restart the controller:

kubectl rollout restart deployment ebs-csi-controller -n kube-system

Fix 3 — Install the EBS CSI Driver (If Missing)

Enterprise Best Practice — EKS Add-on (not Helm):

 # Terraform
 resource "aws_eks_addon" "ebs_csi" {
   cluster_name             = aws_eks_cluster.main.name
   addon_name               = "aws-ebs-csi-driver"
-  # Not configured — driver missing entirely
+  addon_version            = "v1.30.0-eksbuild.1"
+  service_account_role_arn = aws_iam_role.ebs_csi_irsa.arn
+  resolve_conflicts_on_update = "OVERWRITE"
 }

Fix 4 — PVC Referencing Non-Existent StorageClass

 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
   name: postgres-data
 spec:
   accessModes:
     - ReadWriteOnce
   resources:
     requests:
       storage: 20Gi
-  storageClassName: standard
+  storageClassName: gp3-ebs

💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.

(For CI/CD pipeline automation, install via npm i -g stackengine).

package k8s.storageclass

violation[{"msg": msg}] {
  input.review.object.kind == "StorageClass"
  input.review.object.provisioner != "ebs.csi.aws.com"
  msg := sprintf("StorageClass '%v' must use provisioner 'ebs.csi.aws.com', got '%v'",
    [input.review.object.metadata.name, input.review.object.provisioner])
}

violation[{"msg": msg}] {
  input.review.object.kind == "StorageClass"
  input.review.object.volumeBindingMode != "WaitForFirstConsumer"
  msg := "StorageClass must set volumeBindingMode: WaitForFirstConsumer for EBS"
}

# .github/workflows/checkov.yml
- name: Scan Kubernetes manifests
  uses: bridgecrewio/checkov-action@master
  with:
    directory: k8s/
    framework: kubernetes
    check: CKV_K8S_28  # Ensure StorageClass uses approved provisioner
    soft_fail: false

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/antonbabenko/pre-commit-terraform
    hooks:
      - id: terraform_validate
      - id: terraform_tflint

#!/bin/bash
# ebs-smoke-test.sh — run this in your CD pipeline after cluster provisioning
kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-smoke-pvc
  namespace: default
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: gp3-ebs
  resources:
    requests:
      storage: 1Gi
EOF

sleep 30
STATUS=$(kubectl get pvc ebs-smoke-pvc -o jsonpath='{.status.phase}')
if [ "$STATUS" != "Bound" ]; then
  echo "FATAL: EBS CSI smoke test failed. PVC status: $STATUS"
  kubectl describe pvc ebs-smoke-pvc
  exit 1
fi
kubectl delete pvc ebs-smoke-pvc
echo "EBS CSI provisioner: OK"