Calico BGP Peer Connectivity Analyzer

The Incident (What does the error mean?)

Raw output from calicoctl node status:

Calico process is running.

IPv4 BGP status
+---------------+-------------------+-------+----------+-------------+
| PEER ADDRESS  |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+---------------+-------------------+-------+----------+-------------+
| 10.0.1.45     | node-to-node mesh | start | 04:12:33 | Connect     |
| 192.168.10.1  | global            | start | 04:12:33 | Active      |
+---------------+-------------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.

Or from BIRD logs (kubectl logs -n kube-system <calico-node-pod> -c calico-node):

2024-01-15 04:13:01.456 [ERROR] Connection to BGP neighbor 10.0.1.45 failed: Connection refused
2024-01-15 04:13:01.457 [INFO]  BGP session with 10.0.1.45 moved to ACTIVE state

Immediate consequence: BGP is in Connect or Active state — never Established. Calico is not exchanging routes. Every pod on the affected node cannot reach pods on the peer node. kubectl exec cross-node calls hang. Services backed by pods on the unreachable node return connection timeouts. This is a full network partition for affected node pairs.

The Attack Vector / Blast Radius

This is not a flap — Connection refused means TCP/179 is being actively rejected by the remote host's kernel or firewall. This is distinct from a timeout (firewall drop) or a BGP NOTIFICATION (protocol-level rejection).

Cascading failure chain:

1. BIRD on the local node attempts TCP handshake to peer:179.
2. Remote kernel sends RST — port closed or firewall REJECT rule hit.
3. BIRD backs off, retries on exponential timer (up to 120s intervals).
4. No BGP routes exchanged → Calico Felix has no remote workload routes → programs no routes into the kernel routing table.
5. All cross-node pod traffic is unroutable. In vxlan mode this is masked slightly longer; in native BGP mode it's instant.
6. If this is a ToR/upstream router peer (global BGP peer for on-prem), your entire cluster loses external reachability — not just cross-node pod traffic.

In multi-tenant clusters, this silently breaks network policies that depend on cross-node enforcement, creating a false sense of isolation while traffic is simply dropped rather than policy-denied.

Prevention in CI/CD

1. Validate BGPPeer manifests with OPA/Conftest before apply

# policy/bgp_peer.rego
package calico.bgppeer

deny[msg] {
  input.kind == "BGPPeer"
  not input.spec.asNumber
  msg := "BGPPeer must specify asNumber explicitly"
}

deny[msg] {
  input.kind == "BGPPeer"
  input.spec.nodeSelector == "all()"
  msg := "BGPPeer nodeSelector 'all()' is too broad for production — scope to specific nodes"
}

conftest test bgppeer.yaml --policy policy/

2. Smoke-test BGP state in post-deploy pipeline

#!/bin/bash
# ci/verify-bgp.sh — run after Calico config changes
UNESTABLISHED=$(calicoctl node status 2>/dev/null | grep -v Established | grep -c 'start\|Active\|Connect')
if [ "$UNESTABLISHED" -gt 0 ]; then
  echo "FATAL: $UNESTABLISHED BGP peer(s) not established after deploy"
  exit 1
fi
echo "All BGP peers established."

3. Terraform — enforce security group BGP rule presence

# checkov ignore is NOT acceptable here — enforce with policy
# Add to your Checkov custom checks:
# CKV_CUSTOM_BGP_01: "Ensure node security group allows TCP/179 intra-group"

Run Checkov in CI:

checkov -d ./terraform --check CKV_CUSTOM_BGP_01 --compact

4. Alert on BGP session flaps via Prometheus

Calico exports felix_bgp_num_peers and felix_bgp_num_established_peers. Alert when they diverge:

# prometheus-rule.yaml
groups:
- name: calico-bgp
  rules:
  - alert: CalicoBGPPeerNotEstablished
    expr: felix_bgp_num_peers - felix_bgp_num_established_peers > 0
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "BGP peer down on {{ $labels.instance }} — pod routing degraded"