Canal CNI BGP Peering Status Tracker

Threat/Impact Level: HIGH | Downtime Risk: HIGH | Time to Fix: 15–45 mins

TL;DR

• What broke: calico-node pods are NotReady because the embedded bird/bird6 BGP daemon cannot establish peering sessions between nodes, causing cross-node pod traffic to drop silently.
• How to fix it: Unblock TCP/179, correct IP autodetection, fix MTU, or repair the IPPool encapsulation mode — whichever calicoctl node status fingers as the fault.
• Shortcut: Use our Client-Side Sandbox above — drop your calicoctl node status output and FelixConfiguration YAML to auto-generate the corrected config without leaking your cluster topology.

# kubectl get pods -n kube-system | grep calico-node
calico-node-4xk9p   0/1   Running   4   22m
calico-node-r7mnt   0/1   Running   2   22m

# kubectl logs -n kube-system calico-node-4xk9p -c calico-node
2024-01-15 03:42:17.831 [ERROR][8] felix/int_dataplane.go 1394: Failed to connect to BIRD
birdclient: failed to connect to bird socket: dial unix /var/run/calico/bird.ctl: connect: no such file or directory

# calicoctl node status
Calico process is running.

IPv4 BGP status
+---------------+-------------------+-------+----------+-------------+
| PEER ADDRESS  |   PEER TYPE       | STATE |  SINCE   |    INFO     |
+---------------+-------------------+-------+----------+-------------+
| 10.0.1.45     | node-to-node mesh | start | 03:40:11 | Active      |
| 10.0.1.67     | node-to-node mesh | start | 03:40:11 | Active      |
+---------------+-------------------+-------+----------+-------------+

How to Fix It

Step 1: Confirm the actual fault

# On the affected node directly:
sudo calicoctl node status

# Check bird logs:
kubectl logs -n kube-system <calico-node-pod> -c calico-node | grep -i bird

# Verify port 179 reachability between nodes:
nc -zv <peer-node-ip> 179

# Check IP autodetection result:
kubectl get node <node-name> -o jsonpath='{.status.addresses}'

Fix A: TCP/179 Blocked (Most Common in AWS/GCP/Azure)

Basic Fix — Cloud Security Group: Add an inbound rule on your node security group allowing TCP/179 from the node security group itself (self-referencing rule).

# AWS Security Group rule (Terraform)
- # No rule for BGP
+ ingress {
+   from_port       = 179
+   to_port         = 179
+   protocol        = "tcp"
+   self            = true
+   description     = "Canal BGP peering between nodes"
+ }

Also check iptables on the node itself:

iptables -L INPUT -n -v | grep 179
# If blocked, add:
iptables -I INPUT -p tcp --dport 179 -j ACCEPT

Fix B: Wrong Node IP Autodetection

# FelixConfiguration / calico-config ConfigMap in Canal DaemonSet
 apiVersion: v1
 kind: ConfigMap
 metadata:
   name: canal-config
   namespace: kube-system
 data:
-  ip_autodetection_method: "first-found"
+  ip_autodetection_method: "interface=eth0"
+  # OR use CIDR-based detection if interface names vary:
+  # ip_autodetection_method: "cidr=10.0.0.0/8"

After patching the ConfigMap, restart the DaemonSet:

kubectl rollout restart daemonset/canal -n kube-system

Fix C: MTU Misconfiguration (IP-in-IP overhead)

 apiVersion: projectcalico.org/v3
 kind: FelixConfiguration
 metadata:
   name: default
 spec:
-  # mtuIfacePattern not set — Felix auto-detects, often wrong in cloud
+  mtuIfacePattern: "^eth0"
   ipv6Support: false

---
 apiVersion: projectcalico.org/v3
 kind: IPPool
 metadata:
   name: default-ipv4-ippool
 spec:
   cidr: 192.168.0.0/16
-  ipipMode: Never
+  ipipMode: Always        # Required for most cloud providers without BGP support
-  vxlanMode: Never
   natOutgoing: true
+  # Set MTU 20 bytes less than node NIC MTU to account for IP-in-IP header
+  # Node NIC MTU 9001 (jumbo) → set pod MTU to 8981
+  # Node NIC MTU 1500 (standard) → set pod MTU to 1480

Enterprise Best Practice — enforce MTU via FelixConfiguration:

 apiVersion: projectcalico.org/v3
 kind: FelixConfiguration
 metadata:
   name: default
 spec:
+  vxlanMTU: 1450
+  wireguardMTU: 1440
   ipipEnabled: true
+  logSeverityScreen: Info
+  reportingInterval: 30s

💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.

(For CI/CD pipeline automation, install via npm i -g stackengine).

Fix D: Verify BGP Peering After Changes

# Should show 'Established' not 'start'/'Active'
calicoctl node status

# Verify routes are now programmed:
ip route show | grep bird

# Test cross-node pod connectivity:
kubectl run test-pod --image=busybox --rm -it -- ping <pod-ip-on-different-node>

# Add to your cluster bootstrap pipeline
apiVersion: batch/v1
kind: Job
metadata:
  name: bgp-preflight-check
spec:
  template:
    spec:
      hostNetwork: true
      containers:
      - name: check
        image: calico/node:v3.26.0
        command: ["sh", "-c", "calicoctl node status && nc -zv $NODE_PEER_IP 179"]

package canal.bgp

deny[msg] {
  input.kind == "IPPool"
  input.spec.ipipMode == "Never"
  input.spec.vxlanMode == "Never"
  msg := "IPPool must have ipipMode or vxlanMode enabled for cloud deployments. BGP peering without encapsulation requires direct L2 adjacency."
}

# Add to CI step before helm upgrade:
checkov -d ./canal-helm-values/ --check CKV_K8S_28
kube-linter lint canal-manifests/ --config .kube-linter.yaml

# Prometheus alert
- alert: CalicoNodeBGPPeeringDown
  expr: felix_cluster_num_hosts_running - felix_cluster_num_hosts_with_workload > 0
  for: 3m
  labels:
    severity: critical
  annotations:
    summary: "calico-node BGP peering degraded — cross-node routing may be broken"
    runbook: "https://your-wiki/canal-bgp-runbook"

- image: calico/node:latest
+ image: calico/node:v3.26.4  # Pin. Test upgrades in staging. BGP config schema changes between minors.