How to Fix 'Hash Sum Mismatch' in Docker apt-get update (Root Cause + CI/CD Prevention)

Threat/Impact Level: HIGH | Exploitability/Downtime Risk: HIGH | Time to Fix: 5–10 mins

---

TL;DR

• What broke: Docker's cached APT index is stale or a mirror returned a package whose checksum doesn't match the Release file — apt-get update hard-fails and your image build dies.
• How to fix it: Bust the Docker layer cache on the apt-get update step, clean /var/lib/apt/lists/* before and after, and pin to a reliable mirror.
• Fast path: Drop your failing Dockerfile into the Client-Side Sandbox above — it will auto-refactor the RUN block with the correct cache-busting pattern without sending your file anywhere.

---

The Incident (What Does the Error Mean?)

Raw error output from a failing build:

Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1,275 kB]
Err:5 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
  Hash Sum mismatch
  Hashes of expected file:
   - Filesize:1275006 [weak]
   - SHA256:e6b4...
  Hashes of received file:
   - SHA256:a1c9...
E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal/main/binary-amd64/Packages.xz  Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.

Immediate consequence: The RUN apt-get update layer exits non-zero. Docker build stops. Your CI job fails. No image is pushed. If this is a hotfix pipeline, you are now blocked.

The error means APT downloaded an index file from a mirror but its computed SHA-256 (or MD5) does not match the cryptographic hash published in the mirror's Release file. APT refuses to proceed — correctly — because using a tampered or partially-transferred index could result in installing wrong or malicious packages.

---

The Attack Vector / Blast Radius

This is primarily a build reliability failure, but it has a secondary security surface:

1. Stale Docker layer cache (most common): Docker cached the apt-get update layer days or weeks ago. The mirror has since rotated its index. The cached layer holds an old Release file. When a subsequent apt-get install in a separate RUN instruction triggers a re-fetch, the hashes diverge. This is the classic mistake of splitting apt-get update and apt-get install into two RUN commands.

2. Flaky or geo-routed mirror: CDN-backed mirrors (archive.ubuntu.com, deb.debian.org) occasionally serve partially-replicated state during a mirror sync window. A build hitting a mid-sync mirror gets a Packages.xz that doesn't yet match the Release file on the same edge node.

3. Blast radius in CI/CD: Every parallel build job hitting the same mirror at the same time will fail simultaneously. In a Kubernetes-based CI cluster (Tekton, GitHub Actions self-hosted), this can take down your entire build fleet. Retrying without fixing the root cause just hammers the mirror.

4. Security implication: A network-level MITM (e.g., a compromised corporate proxy or a rogue Docker registry mirror) could deliberately serve a mismatched index to force a build failure or, if APT were misconfigured to ignore errors, to inject a backdoored package. APT's hash verification is the last line of defense here.

---

How to Fix It

Basic Fix — Cache-Bust and Clean in a Single RUN

The cardinal rule: apt-get update and apt-get install must be in the same RUN instruction, and /var/lib/apt/lists must be wiped at the end.

- RUN apt-get update
- RUN apt-get install -y curl git build-essential
+ RUN apt-get update --fix-missing && \
+     apt-get install -y --no-install-recommends curl git build-essential && \
+     rm -rf /var/lib/apt/lists/*

If the build is still failing due to a mid-sync mirror, force a clean fetch by adding --no-cache to the docker build command:

- docker build -t myapp:latest .
+ docker build --no-cache -t myapp:latest .

Or invalidate only the APT layer by bumping a build arg:

+ ARG APT_CACHE_DATE=2024-01-01
  RUN apt-get update && apt-get install -y ...

Change APT_CACHE_DATE in your CI invocation to force re-execution of that layer only.

---

Enterprise Best Practice — Mirror Pinning + Retry Logic + Verified Sources

For production Dockerfiles that must be reproducible and resilient:

- FROM ubuntu:22.04
- RUN apt-get update && apt-get install -y curl

+ FROM ubuntu:22.04
+
+ # Pin to a stable, well-maintained mirror. Replace with your regional mirror.
+ RUN sed -i 's|http://archive.ubuntu.com|http://us.archive.ubuntu.com|g' /etc/apt/sources.list
+
+ # Single atomic RUN: update, install, clean. Never split these.
+ RUN apt-get update -o Acquire::Retries=3 && \
+     apt-get install -y --no-install-recommends \
+         curl \
+         ca-certificates && \
+     apt-get clean && \
+     rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

Key flags explained:

• Acquire::Retries=3 — APT will retry failed fetches 3 times before giving up, handling transient mirror sync windows.
• --no-install-recommends — reduces attack surface and image size.
• apt-get clean && rm -rf /var/lib/apt/lists/* — ensures the layer contains zero cached index data, preventing stale-hash issues in child images.

For air-gapped or regulated environments, host a private APT mirror (Artifactory, Nexus, apt-mirror) and point your base images at it. This eliminates public mirror flakiness entirely.

---

💡 Tired of pasting proprietary configs into ChatGPT? Generic AI tools log your company's ARNs, DB strings, and private keys. StackEngine is a zero-backend, pure Client-Side WASM utility. Drop your failing config into the sandbox above. We redact your secrets locally in the browser and auto-generate the refactored code using your own API key.

---

Prevention in CI/CD

1. Hadolint in your pipeline (catches the split RUN anti-pattern):

# .github/workflows/lint.yml
- name: Lint Dockerfile
  uses: hadolint/[email protected]
  with:
    dockerfile: Dockerfile
    failure-threshold: warning

Hadolint rule DL3009 flags apt-get update not paired with apt-get install in the same layer. Rule DL3008 enforces pinned package versions.

2. Enforce with OPA/Conftest against Dockerfile policies:

# policy/dockerfile.rego
package dockerfile

deny[msg] {
  input[i].Cmd == "run"
  val := input[i].Value
  contains(val[_], "apt-get update")
  not contains(val[_], "rm -rf /var/lib/apt/lists")
  msg := "apt-get update RUN block must clean /var/lib/apt/lists/* in the same layer"
}

3. Scheduled base image rebuilds: Do not rely on Docker's layer cache for base images older than 7 days. Use a weekly scheduled pipeline that rebuilds all base images with --no-cache. This keeps your APT index fresh and catches upstream CVEs.

4. Private mirror for reproducibility: In enterprise environments, route all APT traffic through Artifactory or Nexus with a cron-synced upstream. Pin your sources.list in a controlled base image layer. Zero dependency on public mirror availability during builds.