Etcd Leader Election Partition Resolver

The Incident (What Does the Error Mean?)

Raw etcd log output from the partitioned node:

raft2025/01/15 03:12:44 INFO: raft.node: 8e9e05c52164694d lost leader 8e9e05c52164694d at term 42
raft2025/01/15 03:12:44 INFO: raft.node: 8e9e05c52164694d is starting a new election at term 42
raft2025/01/15 03:12:44 WARN: etcdserver: failed to send out heartbeat on time; took too long, leader is overloaded
raft2025/01/15 03:12:44 WARN: etcdserver: server is likely overloaded
{"level":"warn","ts":"2025-01-15T03:12:45.112Z","caller":"etcdserver/server.go:1084","msg":"failed to reach the peer URL","peer-id":"a8266ecf031671f3","peer-url":"https://10.0.1.12:2380"}
{"level":"warn","ts":"2025-01-15T03:12:47.001Z","caller":"raft/raft.go:900","msg":"no leader elected"}

Immediate consequence: The Kubernetes API server returns etcdserver: leader changed or hangs on all mutating requests (kubectl apply, kubectl delete, pod scheduling). The cluster enters a read-only degraded state. No new pods schedule. Existing workloads continue running, but the control plane is effectively dead.

The Attack Vector / Blast Radius

This is a quorum failure, not a security exploit — but the blast radius is equivalent to a full control plane outage.

Cascade chain:

1. Network partition or disk I/O spike causes heartbeat packets to exceed --election-timeout on follower nodes.
2. Followers declare the leader dead and start a new election. If the partitioned node can still see some peers but not all, you get a split-brain election loop — no candidate achieves (N/2)+1 votes.
3. kube-apiserver loses its etcd watch stream. All LIST/WATCH calls to etcd block or return errors.
4. kube-scheduler and kube-controller-manager lose leader leases. They stop functioning.
5. CoreDNS, ingress controllers, HPA — anything that writes to the API server — queues up or fails.
6. If the partition persists >5 minutes, etcd WAL can diverge. A naive etcdctl member remove + rejoin without a snapshot restore will corrupt the cluster.

Compounding factor: On cloud VMs, a noisy-neighbor disk I/O spike on the etcd node's underlying host causes fsync latency to spike above the election timeout. This is the #1 cause of spurious etcd leader elections in AWS/GCP environments. etcd is brutally sensitive to disk latency — p99 fsync must stay under 10ms.

Prevention in CI/CD

1. Etcd Health Gate in CI

Add an etcd health check to your cluster provisioning pipeline before any control plane operation:

# In your Terraform/Ansible post-apply step
etcdctl endpoint health --cluster 2>&1 | grep -v "is healthy" && exit 1

2. Prometheus Alerting Rules

Deploy these alerts. If you don't have these firing before a partition happens, you're flying blind:

# prometheus-etcd-alerts.yaml
groups:
- name: etcd.critical
  rules:
  - alert: EtcdNoLeader
    expr: etcd_server_has_leader == 0
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "Etcd cluster has no leader — control plane writes halted"

  - alert: EtcdHighFsyncDuration
    expr: histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) > 0.01
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Etcd WAL fsync p99 > 10ms — leader election risk"

  - alert: EtcdMemberCommunicationSlow
    expr: histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket[5m])) > 0.15
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Etcd peer RTT p99 > 150ms — election timeout risk"

3. Infrastructure Policy (OPA/Gatekeeper)

Enforce etcd node placement on dedicated instance types with local NVMe:

# opa-etcd-placement.rego
package kubernetes.admission

deny[msg] {
  input.request.kind.kind == "Node"
  input.request.object.metadata.labels["node-role.kubernetes.io/etcd"] == "true"
  not input.request.object.metadata.labels["node.kubernetes.io/instance-type"] in {"i3.xlarge", "i3en.xlarge", "n2-highmem-4"}
  msg := "Etcd nodes must run on NVMe-backed instance types to guarantee fsync latency SLA"
}

4. Terraform Validation

# Enforce etcd nodes on dedicated AZs — never co-locate all 3 in one AZ
resource "aws_instance" "etcd" {
  count             = 3
  availability_zone = element(["us-east-1a", "us-east-1b", "us-east-1c"], count.index)
  instance_type     = "i3.xlarge"  # Local NVMe mandatory

  lifecycle {
    prevent_destroy = true  # Never destroy etcd nodes without snapshot backup
  }
}

Run checkov -f etcd.tf --check CKV_AWS_8 to validate encryption and placement before apply.