Longhorn Volume Replica Attachment Auditor

The Incident (What Does the Error Mean?)

Raw error surface — typically seen in kubectl describe pod or Longhorn Manager logs:

Warning  FailedAttachVolume  kubelet  AttachVolume.Attach failed for volume "pvc-a1b2c3d4":
  rpc error: code = Internal desc = failed to attach volume: volume controller failed to attach,
  no healthy replica available, replica count: 0

And in Longhorn Manager (kubectl logs -n longhorn-system -l app=longhorn-manager):

time="..." level=error msg="failed to attach volume pvc-a1b2c3d4"
  reason="replica count is 0, no replica available for scheduling"

Immediate consequence: Every pod mounting this PVC is stuck in ContainerCreating or Pending. StatefulSets, databases, and any workload with a volumeMount referencing this PVC are fully down. Kubernetes will not reschedule the pod to another node — it will sit and retry indefinitely.

The Attack Vector / Blast Radius

This is not a transient blip. Zero replicas means Longhorn has no copy of the volume data it can serve. How you got here matters:

Scenario A — Node mass eviction or cloud provider spot interruption: All nodes hosting replicas were terminated simultaneously. Longhorn marks replicas Error. If replicaCount: 1 was set (anti-pattern), you have zero redundancy and zero recovery path without a backup.

Scenario B — Disk pressure / node taint: Nodes were tainted (node.longhorn.io/create-default-disk: false) or disks were manually disabled in the Longhorn UI. The scheduler cannot place replicas. Existing replicas on those disks go Stopped.

Scenario C — Longhorn upgrade gone wrong: A manager rollout left replica CRDs in an inconsistent state. The engine image mismatch causes replicas to fail health checks.

Blast radius: In a StatefulSet with 10 pods across 10 PVCs, if your storage class has numberOfReplicas: 1 and a single AZ goes down, all 10 volumes hit zero replicas simultaneously. Your entire stateful tier is offline. This cascades to connection pool exhaustion in upstream services within seconds.

Prevention in CI/CD

OPA/Gatekeeper Policy — Block single-replica StorageClasses

package longhorn.storagepolicy

deny[msg] {
  input.kind == "StorageClass"
  input.provisioner == "driver.longhorn.io"
  to_number(input.parameters.numberOfReplicas) < 2
  msg := sprintf(
    "StorageClass '%v' sets numberOfReplicas < 2. Minimum is 2 for non-dev workloads.",
    [input.metadata.name]
  )
}

Checkov / Helm pre-flight check

# Add to your CI pipeline before helm upgrade
checkov -d ./helm/values.yaml \
  --check CKV_LONGHORN_REPLICA_COUNT \
  --compact

Longhorn Backup Policy (non-negotiable for production)

# Recurring job — snapshot every 1h, backup every 6h, retain 14 days
apiVersion: longhorn.io/v1beta2
kind: RecurringJob
metadata:
  name: production-backup
  namespace: longhorn-system
spec:
  cron: "0 */6 * * *"
  task: backup
  groups:
    - default
  retain: 56
  concurrency: 2

Tag all production PVCs with recurring-job-group.longhorn.io/default: enabled so this job picks them up automatically. Without this, zero replicas + no backup = unrecoverable data loss.

Alerting — Don't find out from users

# Prometheus alert rule
- alert: LonghornVolumeReplicaCountCritical
  expr: longhorn_volume_robustness == 3  # 3 = Faulted
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Longhorn volume {{ $labels.volume }} is Faulted — replica count may be zero"
    runbook: "https://your-wiki/longhorn-replica-recovery"