Docker Swarm Overlay Split Brain Analyzer

Threat/Impact Level: CRITICAL | Exploitability/Downtime Risk: HIGH | Time to Fix: 20–45 mins

TL;DR

• What broke: The Swarm overlay network's VXLAN/gossip state diverged between nodes — containers on different hosts are isolated, service discovery is dead, and inter-service traffic is silently dropped.
• How to fix it: Force-remove the stale overlay network, purge dangling VXLAN interfaces, re-advertise node state, and recreate the network with explicit subnet pinning.
• CTA: Use our Client-Side Sandbox above to paste your docker network inspect output and get auto-generated remediation commands without leaking your config to a third-party AI.

$ docker exec -it service_replica_1 ping 10.0.1.5
ping: connect: Network is unreachable

$ docker network inspect my_overlay
[
  {
    "Name": "my_overlay",
    "Id": "q7k2p...",
    "Peers": [
      { "Name": "node1", "IP": "192.168.1.10" }
    ]
    // node2, node3 MISSING from Peers list
  }
]

How to Fix It

Step 0: Confirm the Split-Brain

# Run on EACH node — compare Peers lists
docker network inspect <overlay_name> --format '{{json .Peers}}' | python3 -m json.tool

# Check VXLAN interfaces exist
ip -d link show type vxlan

# Check if the FDB (forwarding database) has entries for other nodes
bridge fdb show dev vx-<network_short_id>

If bridge fdb shows no entries for remote node MACs, the VTEP has no path. This confirms split-brain.

Basic Fix: Force Network Reconciliation

- # Leaving the stale overlay attached and hoping swarm self-heals
- # (it will not — gossip convergence does not rebuild VXLAN FDB automatically)

+ # Step 1: Scale down all services using the broken network
+ docker service scale my_service=0
+
+ # Step 2: On ALL worker nodes, remove dangling VXLAN interfaces
+ # Find the vxlan interface name first:
+ ip link show type vxlan
+ # Then delete it:
+ sudo ip link delete vx-<short_network_id>
+
+ # Step 3: Remove the overlay network from the manager
+ docker network rm my_overlay
+
+ # Step 4: Recreate with explicit, pinned subnet
+ docker network create \
+   --driver overlay \
+   --subnet 10.0.9.0/24 \
+   --opt com.docker.network.driver.mtu=1450 \
+   my_overlay
+
+ # Step 5: Redeploy
+ docker service scale my_service=3

Enterprise Best Practice: Prevent Gossip Desync at the Node Level

- # Default swarm init — no advertise-addr, no data-path-addr separation
- docker swarm init

+ # Explicitly separate control plane and data plane traffic
+ # Control plane (gossip, raft) on management NIC
+ # Data plane (VXLAN encap) on high-throughput NIC
+ docker swarm init \
+   --advertise-addr eth0:2377 \
+   --data-path-addr eth1 \
+   --data-path-port 4789
+
+ # On worker join:
+ docker swarm join \
+   --advertise-addr eth0 \
+   --data-path-addr eth1 \
+   --token <worker-token> \
+   <manager-ip>:2377

- # No MTU configuration in compose — inherits host default (1500)
- networks:
-   my_overlay:
-     driver: overlay

+ # Pin MTU explicitly — 1450 for standard VXLAN, 1400 if running over IPSec/WireGuard
+ networks:
+   my_overlay:
+     driver: overlay
+     driver_opts:
+       com.docker.network.driver.mtu: "1450"
+     ipam:
+       config:
+         - subnet: 10.0.9.0/24

Nuclear Option (Full Node Rejoin)

If the above doesn't resolve it — the node's swarm state is corrupted:

# On the affected worker:
docker swarm leave --force
sudo systemctl stop docker
sudo rm -rf /var/lib/docker/swarm
sudo systemctl start docker
docker swarm join --token <worker-token> <manager>:2377

💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.

(For CI/CD pipeline automation, install via npm i -g stackengine).

# Add to your deploy pipeline before any docker stack deploy
NODE_MTU=$(ip link show eth0 | grep -oP 'mtu \K[0-9]+')
if [ "$NODE_MTU" -lt 1500 ]; then
  echo "ERROR: Underlay MTU $NODE_MTU too low for default overlay. Set overlay MTU to $((NODE_MTU - 50))"
  exit 1
fi

# policy/overlay_mtu.rego
package docker.compose

deny[msg] {
  net := input.networks[_]
  net.driver == "overlay"
  not net.driver_opts["com.docker.network.driver.mtu"]
  msg := sprintf("Overlay network '%v' must explicitly set MTU via driver_opts", [net])
}

# In CI:
conftest test docker-compose.yml --policy policy/

#!/bin/bash
# Run on manager — alert if any node's overlay peer count drops below expected
EXPECTED_PEERS=$(docker node ls --filter role=worker -q | wc -l)
ACTUAL_PEERS=$(docker network inspect my_overlay --format '{{len .Peers}}')

if [ "$ACTUAL_PEERS" -lt "$EXPECTED_PEERS" ]; then
  echo "ALERT: Overlay split-brain detected. Expected $EXPECTED_PEERS peers, found $ACTUAL_PEERS"
  # Push to PagerDuty/Alertmanager here
fi

# In your node base image Dockerfile
FROM ubuntu:22.04
RUN apt-get install -y docker-ce=5:24.0.7* linux-image-5.15.0-91-generic