Fixing 'unexpected EOF' in docker logs --tail on Large Log Files

Threat/Impact Level: HIGH | Downtime Risk: HIGH | Time to Fix: 15 mins

TL;DR

• What broke: Docker's default json-file log driver writes unbounded JSON log files. On files exceeding ~1–2GB, the log reader hits a partial JSON token mid-read and throws unexpected EOF, returning zero or truncated lines.
• How to fix it: Enforce max-size and max-file log rotation on the daemon or per-container, then manually truncate or rotate the bloated log file without restarting the container.
• Fast path: Use our Client-Side Sandbox below to auto-refactor your daemon.json or docker-compose.yml with correct log rotation limits applied.

---

The Incident (What does the error mean?)

Raw error output from the Docker CLI or daemon:

$ docker logs --tail 100 my_app_container
unexpected EOF

Or via the API:

Error response from daemon: invalid character '\x00' looking for beginning of value

Docker's json-file driver stores every log line as a JSON object in a flat file located at:

/var/lib/docker/containers/<container-id>/<container-id>-json.log

When this file grows to several gigabytes, two failure modes trigger:

1. Partial JSON token at tail boundary — the --tail reader seeks from the end of the file and lands mid-JSON-object, producing unexpected EOF.
2. Null byte corruption — disk pressure or a crash during a write leaves \x00 padding, making the JSON parser abort immediately.

Immediate consequence: Operators are blind. You cannot read recent logs from a running production container without a restart or external log shipper.

---

The Attack Vector / Blast Radius

This is a silent operational failure, not a CVE — but the blast radius in production is severe:

• Disk exhaustion cascade: A single verbose container (e.g., a Java app with DEBUG logging) can fill /var/lib/docker to 100%, which kills the Docker daemon itself, taking down every container on the host simultaneously.
• Incident blindness: The exact moment you need logs most — during an outage — is when this error surfaces. docker logs returns nothing.
• OOM interaction: Large log files are often co-located with memory pressure. The kernel may OOM-kill the Docker daemon's log read process, compounding the failure.
• CI/CD pipeline corruption: Build agents running long jobs accumulate multi-GB logs. A failed docker logs call in a pipeline script exits non-zero and marks a passing build as failed.
• Monitoring gap: If you rely on docker logs as a sidecar scrape path (e.g., Promtail, Filebeat), the reader process will crash-loop on the corrupted file, creating a log ingestion gap in your SIEM or observability stack.

---

How to Fix It (The Solution)

Step 1 — Identify the bloated log file

# Find the container log file and its size
CONTAINER_ID=$(docker inspect --format='{{.Id}}' my_app_container)
ls -lh /var/lib/docker/containers/${CONTAINER_ID}/${CONTAINER_ID}-json.log

If this file is >500MB, you have found your problem.

---

Basic Fix — Truncate the log file without restarting the container

Do NOT use rm. The file descriptor is held open by the Docker daemon. Use truncation:

# Safe truncation — zeroes the file, keeps the fd open
truncate -s 0 /var/lib/docker/containers/${CONTAINER_ID}/${CONTAINER_ID}-json.log

# Verify
docker logs --tail 10 my_app_container

This is a live fix. The container keeps running. New log lines appear immediately after truncation.

---

Enterprise Best Practice — Enforce log rotation at the daemon level

This must be set in /etc/docker/daemon.json. Without it, every container on every host is vulnerable.

# /etc/docker/daemon.json

-{
-  "log-driver": "json-file"
-}

+{
+  "log-driver": "json-file",
+  "log-opts": {
+    "max-size": "100m",
+    "max-file": "5",
+    "compress": "true"
+  }
+}

Apply without full daemon restart on systemd hosts:

sudo systemctl reload docker
# Note: reload applies daemon.json changes to NEW containers only.
# Existing containers require recreation to inherit new log opts.

---

Per-container override in docker-compose.yml

For containers that cannot wait for a daemon reload:

# docker-compose.yml

 services:
   app:
     image: my_app:latest
+    logging:
+      driver: "json-file"
+      options:
+        max-size: "50m"
+        max-file: "10"
+        compress: "true"

---

Nuclear option — Switch to a non-file log driver

For high-throughput services (>10k lines/sec), the json-file driver is the wrong tool entirely:

-  "log-driver": "json-file"
+  "log-driver": "local"

The local driver uses protobuf encoding instead of JSON, is 2–3x more space-efficient, and handles large files without the JSON parse boundary issue. Caveat: docker logs still works, but third-party tools expecting raw JSON files will break.

---

💡 Tired of pasting proprietary configs into ChatGPT? Generic AI tools log your company's ARNs, DB strings, and private keys. StackEngine is a zero-backend, pure Client-Side WASM utility. Drop your failing config into the sandbox above. We redact your secrets locally in the browser and auto-generate the refactored code using your own API key.

---

Prevention in CI/CD

1. Enforce daemon.json via configuration management

Never let a Docker host exist without enforced log rotation. Use Ansible, Chef, or cloud-init:

# Ansible task
- name: Enforce Docker log rotation
  copy:
    dest: /etc/docker/daemon.json
    content: |
      {
        "log-driver": "json-file",
        "log-opts": {
          "max-size": "100m",
          "max-file": "5"
        }
      }
  notify: reload docker

2. Checkov policy for docker-compose files

pip install checkov
checkov -d . --check CKV_DOCKER_7
# CKV_DOCKER_7 checks for log rotation options on services

3. OPA/Conftest policy for daemon.json in Terraform

If you provision EC2/GCE instances with user-data that writes daemon.json, gate it in your Terraform pipeline:

# policy/docker_logging.rego
package docker.daemon

deny[msg] {
  input.log_opts["max-size"] == ""
  msg := "Docker daemon.json must define log-opts.max-size"
}

conftest test daemon.json --policy policy/

4. Disk usage alerting

Add a Prometheus alert before the disk fills:

# alertmanager rule
- alert: DockerLogDiskPressure
  expr: (node_filesystem_avail_bytes{mountpoint="/var/lib/docker"} / node_filesystem_size_bytes{mountpoint="/var/lib/docker"}) < 0.15
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Docker log volume >85% full on {{ $labels.instance }}"

5. Logrotate as a fallback

For hosts you cannot modify the daemon on immediately:

# /etc/logrotate.d/docker-logs
/var/lib/docker/containers/*/*.log {
  rotate 5
  copytruncate
  daily
  compress
  missingok
  delaycompress
  maxsize 100M
}

copytruncate is mandatory here — it copies then truncates rather than moving the file, preserving the open file descriptor held by dockerd.