Docker MTU Mismatch Packet Drop Fix

The Incident (What Does the Error Mean?)

You won't get a clean error. This is what makes it brutal. Symptoms look like:

# curl from inside container to external service hangs on large payloads
curl: (18) transfer closed with outstanding read data remaining

# ping works, but TCP sessions stall
ping 8.8.8.8   # succeeds
curl https://api.example.com  # hangs at TLS handshake or after headers

# tcpdump on docker0 shows retransmits
tcpdump -i docker0 -n 'tcp and (tcp[tcpflags] & tcp-syn != 0)'
# ... endless SYN-ACK retransmits for large-window packets

Immediate consequence: Any TCP session transferring payloads larger than the underlay MTU minus encapsulation overhead will stall or fail. This kills HTTPS API calls, database connections, and inter-service gRPC streams — silently, intermittently, and only under load. On-call engineers waste hours chasing application bugs that are actually a 50-byte MTU delta.

The Attack Vector / Blast Radius

This is a cascading infrastructure failure, not a one-container problem.

Why it's invisible: Most cloud VPCs (AWS, GCP, Azure) disable ICMP Type 3 Code 4 ("Fragmentation Needed") responses at the security group or VPC router level. Path MTU Discovery (PMTUD) breaks. Packets are simply dropped. TCP retransmits forever. Your application sees a hung connection.

Blast radius by environment:

Cascading failure path:

1. Container MTU 1500 > tunnel MTU 1450
2. Large packet hits docker0, exits host NIC, gets dropped at underlay router
3. No ICMP fragmentation-needed returned (cloud firewall blocks it)
4. TCP sender never learns the path MTU, keeps sending oversized segments
5. Connection hangs → application timeout → retry storm → upstream service rate-limits your IPs → full service degradation

Prevention in CI/CD

The goal: Catch MTU misconfiguration before it reaches production.

1. Checkov — Scan `daemon.json` in IaC pipelines

Checkov doesn't have a built-in Docker MTU rule, so write a custom check:

# checkov/custom_checks/docker_mtu.py
from checkov.common.models.enums import CheckResult, CheckCategories
from checkov.json_doc.base_json_check import BaseJsonCheck

class DockerMTUCheck(BaseJsonCheck):
    def __init__(self):
        name = "Ensure Docker daemon MTU is set below 1500 for overlay environments"
        id = "CKV_DOCKER_MTU_001"
        super().__init__(name=name, id=id, categories=[CheckCategories.NETWORKING], supported_entities=['*'])

    def get_resource_id(self, conf):
        return "daemon.json"

    def scan_resource_conf(self, conf):
        mtu = conf.get("mtu", 1500)
        if isinstance(mtu, int) and mtu <= 1450:
            return CheckResult.PASSED
        return CheckResult.FAILED

2. CI Pipeline Gate (GitHub Actions)

# .github/workflows/docker-config-lint.yml
jobs:
  lint-docker-config:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Validate daemon.json MTU
        run: |
          MTU=$(jq '.mtu // 1500' infra/daemon.json)
          if [ "$MTU" -gt 1450 ]; then
            echo "ERROR: daemon.json MTU ($MTU) exceeds safe threshold for overlay networks. Set to <= 1450."
            exit 1
          fi

3. Runtime Monitoring — Alert on MTU Mismatch

# Add to your node bootstrap / cloud-init script
HOST_MTU=$(ip link show eth0 | grep -oP 'mtu \K[0-9]+')
DOCKER_MTU=$(docker network inspect bridge --format '{{json .Options}}' | jq -r '["com.docker.network.driver.mtu"] // "1500"')

if [ "$DOCKER_MTU" -ge "$HOST_MTU" ]; then
  echo "WARN: docker0 MTU ($DOCKER_MTU) >= host MTU ($HOST_MTU). Packet drops likely."
  # Send to PagerDuty / Datadog / Slack webhook here
fi

4. Terraform — Enforce MTU in User Data

# terraform/modules/ec2-docker-host/main.tf
 resource "aws_instance" "docker_host" {
   user_data = <<-EOF
     #!/bin/bash
     cat > /etc/docker/daemon.json <<'DOCKERCFG'
     {
-      "log-driver": "json-file"
+      "log-driver": "json-file",
+      "mtu": 1450
     }
     DOCKERCFG
     systemctl restart docker
   EOF
 }

Enforce this with a Sentinel or OPA policy in your Terraform Cloud workspace to reject any daemon.json template missing the mtu key when the instance is tagged overlay: true.