Docker Tail Unexpected EOF Large Log Tool

The Incident (What does the error mean?)

Raw error output from the Docker CLI or daemon:

$ docker logs --tail 100 my_app_container
unexpected EOF

Or via the API:

Error response from daemon: invalid character '\x00' looking for beginning of value

Docker's json-file driver stores every log line as a JSON object in a flat file located at:

/var/lib/docker/containers/<container-id>/<container-id>-json.log

When this file grows to several gigabytes, two failure modes trigger:

1. Partial JSON token at tail boundary — the --tail reader seeks from the end of the file and lands mid-JSON-object, producing unexpected EOF.
2. Null byte corruption — disk pressure or a crash during a write leaves \x00 padding, making the JSON parser abort immediately.

Immediate consequence: Operators are blind. You cannot read recent logs from a running production container without a restart or external log shipper.

The Attack Vector / Blast Radius

This is a silent operational failure, not a CVE — but the blast radius in production is severe:

• Disk exhaustion cascade: A single verbose container (e.g., a Java app with DEBUG logging) can fill /var/lib/docker to 100%, which kills the Docker daemon itself, taking down every container on the host simultaneously.
• Incident blindness: The exact moment you need logs most — during an outage — is when this error surfaces. docker logs returns nothing.
• OOM interaction: Large log files are often co-located with memory pressure. The kernel may OOM-kill the Docker daemon's log read process, compounding the failure.
• CI/CD pipeline corruption: Build agents running long jobs accumulate multi-GB logs. A failed docker logs call in a pipeline script exits non-zero and marks a passing build as failed.
• Monitoring gap: If you rely on docker logs as a sidecar scrape path (e.g., Promtail, Filebeat), the reader process will crash-loop on the corrupted file, creating a log ingestion gap in your SIEM or observability stack.

Prevention in CI/CD

1. Enforce daemon.json via configuration management

Never let a Docker host exist without enforced log rotation. Use Ansible, Chef, or cloud-init:

# Ansible task
- name: Enforce Docker log rotation
  copy:
    dest: /etc/docker/daemon.json
    content: |
      {
        "log-driver": "json-file",
        "log-opts": {
          "max-size": "100m",
          "max-file": "5"
        }
      }
  notify: reload docker

2. Checkov policy for docker-compose files

pip install checkov
checkov -d . --check CKV_DOCKER_7
# CKV_DOCKER_7 checks for log rotation options on services

3. OPA/Conftest policy for daemon.json in Terraform

If you provision EC2/GCE instances with user-data that writes daemon.json, gate it in your Terraform pipeline:

# policy/docker_logging.rego
package docker.daemon

deny[msg] {
  input.log_opts["max-size"] == ""
  msg := "Docker daemon.json must define log-opts.max-size"
}

conftest test daemon.json --policy policy/

4. Disk usage alerting

Add a Prometheus alert before the disk fills:

# alertmanager rule
- alert: DockerLogDiskPressure
  expr: (node_filesystem_avail_bytes{mountpoint="/var/lib/docker"} / node_filesystem_size_bytes{mountpoint="/var/lib/docker"}) < 0.15
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Docker log volume >85% full on {{ $labels.instance }}"

5. Logrotate as a fallback

For hosts you cannot modify the daemon on immediately:

# /etc/logrotate.d/docker-logs
/var/lib/docker/containers/*/*.log {
  rotate 5
  copytruncate
  daily
  compress
  missingok
  delaycompress
  maxsize 100M
}

copytruncate is mandatory here — it copies then truncates rather than moving the file, preserving the open file descriptor held by dockerd.