Docker Apt-Get Hash Sum Matcher

The Incident (What Does the Error Mean?)

Raw error output from a failing build:

Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1,275 kB]
Err:5 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
  Hash Sum mismatch
  Hashes of expected file:
   - Filesize:1275006 [weak]
   - SHA256:e6b4...
  Hashes of received file:
   - SHA256:a1c9...
E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal/main/binary-amd64/Packages.xz  Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.

Immediate consequence: The RUN apt-get update layer exits non-zero. Docker build stops. Your CI job fails. No image is pushed. If this is a hotfix pipeline, you are now blocked.

The error means APT downloaded an index file from a mirror but its computed SHA-256 (or MD5) does not match the cryptographic hash published in the mirror's Release file. APT refuses to proceed — correctly — because using a tampered or partially-transferred index could result in installing wrong or malicious packages.

The Attack Vector / Blast Radius

This is primarily a build reliability failure, but it has a secondary security surface:

1. Stale Docker layer cache (most common): Docker cached the apt-get update layer days or weeks ago. The mirror has since rotated its index. The cached layer holds an old Release file. When a subsequent apt-get install in a separate RUN instruction triggers a re-fetch, the hashes diverge. This is the classic mistake of splitting apt-get update and apt-get install into two RUN commands.

2. Flaky or geo-routed mirror: CDN-backed mirrors (archive.ubuntu.com, deb.debian.org) occasionally serve partially-replicated state during a mirror sync window. A build hitting a mid-sync mirror gets a Packages.xz that doesn't yet match the Release file on the same edge node.

3. Blast radius in CI/CD: Every parallel build job hitting the same mirror at the same time will fail simultaneously. In a Kubernetes-based CI cluster (Tekton, GitHub Actions self-hosted), this can take down your entire build fleet. Retrying without fixing the root cause just hammers the mirror.

4. Security implication: A network-level MITM (e.g., a compromised corporate proxy or a rogue Docker registry mirror) could deliberately serve a mismatched index to force a build failure or, if APT were misconfigured to ignore errors, to inject a backdoored package. APT's hash verification is the last line of defense here.

Prevention in CI/CD

1. Hadolint in your pipeline (catches the split RUN anti-pattern):

# .github/workflows/lint.yml
- name: Lint Dockerfile
  uses: hadolint/hadolint-action@v3.1.0
  with:
    dockerfile: Dockerfile
    failure-threshold: warning

Hadolint rule DL3009 flags apt-get update not paired with apt-get install in the same layer. Rule DL3008 enforces pinned package versions.

2. Enforce with OPA/Conftest against Dockerfile policies:

# policy/dockerfile.rego
package dockerfile

deny[msg] {
  input[i].Cmd == "run"
  val := input[i].Value
  contains(val[_], "apt-get update")
  not contains(val[_], "rm -rf /var/lib/apt/lists")
  msg := "apt-get update RUN block must clean /var/lib/apt/lists/* in the same layer"
}

3. Scheduled base image rebuilds: Do not rely on Docker's layer cache for base images older than 7 days. Use a weekly scheduled pipeline that rebuilds all base images with --no-cache. This keeps your APT index fresh and catches upstream CVEs.

4. Private mirror for reproducibility: In enterprise environments, route all APT traffic through Artifactory or Nexus with a cron-synced upstream. Pin your sources.list in a controlled base image layer. Zero dependency on public mirror availability during builds.