Fix: `docker logs -f` Shows Nothing After Container Restart with JSON-File Log Rotation

Threat/Impact Level: HIGH | Downtime Risk: HIGH | Time to Fix: 10 mins

---

TL;DR

• What broke: After container restart, docker logs -f returns empty because the active log pointer references a rotated-out file handle that no longer exists at the expected inode.
• How to fix it: Set max-file to retain enough rotated files AND ensure the container's log symlink is valid post-restart; force log re-initialization by removing stale *-json.log files before restart.
• Shortcut: Use our Client-Side Sandbox above to paste your daemon.json or docker-compose.yml and auto-generate the corrected log driver config.

---

The Incident (What Does the Error Mean?)

You run:

docker logs -f my_container
# Returns immediately with no output. Zero lines.

Or:

docker logs --tail 100 my_container
# (empty)

Under /var/lib/docker/containers/<container-id>/, you find:

<container-id>-json.log        # 0 bytes or missing
<container-id>-json.log.1     # the actual data, rotated out
<container-id>-json.log.2

What happened: The json-file driver rotated the active log file when it hit max-size. On container restart, Docker re-opens the log file descriptor pointing to *-json.log. If that file was truncated to 0 bytes or recreated empty by the rotation mechanism during the restart window, the daemon's log reader finds nothing. The rotated .log.1, .log.2 files are not surfaced by docker logs — only the current *-json.log is read.

---

The Blast Radius

This is not cosmetic. The consequences in production:

• Silent application failures. Your app is throwing panics, OOM errors, or segfaults. You see nothing in docker logs. You're flying blind during an incident.
• Alerting pipelines break. Log shippers (Fluentd, Filebeat, Promtail) configured to tail docker logs or the raw JSON file get a 0-byte file and stop shipping. Your Grafana/Loki/Splunk dashboards go dark exactly when you need them.
• Post-mortem data is gone. If max-file: "1" (the default), rotation overwrites the single backup. Evidence of the crash is deleted.
• Cascading restart loops hide the root cause. A crashlooping container keeps rotating and truncating. Every restart wipes the previous log. You can never see why it's crashing.

---

How to Fix It

Basic Fix — Correct daemon.json Log Options

The immediate problem is almost always an under-configured max-file combined with max-size being too small.

# /etc/docker/daemon.json
{
-  "log-driver": "json-file",
-  "log-opts": {
-    "max-size": "10m"
-  }
+  "log-driver": "json-file",
+  "log-opts": {
+    "max-size": "50m",
+    "max-file": "5",
+    "compress": "true"
+  }
}

Apply without full daemon restart (Docker 20.10+):

sudo kill -SIGHUP $(pidof dockerd)

For containers already in a broken state — recover the lost log pointer:

# Stop the container cleanly
docker stop my_container

# Check what's in the log directory
ls -lah /var/lib/docker/containers/$(docker inspect --format='{{.Id}}' my_container)/

# If *-json.log is 0 bytes but .log.1 has data, manually recover:
cp /var/lib/docker/containers/<id>/<id>-json.log.1 /var/lib/docker/containers/<id>/<id>-json.log

# Restart
docker start my_container
docker logs -f my_container

---

Enterprise Best Practice — Per-Service Log Config in Compose

Never rely solely on daemon-level defaults. Override per service so critical workloads retain more history.

# docker-compose.yml
services:
  api:
    image: my-api:latest
+   logging:
+     driver: "json-file"
+     options:
+       max-size: "100m"
+       max-file: "10"
+       compress: "true"
+       labels: "service_name,environment"
-   # No logging block — inherits daemon default of max-size=0 (unlimited) or misconfigured global

  worker:
    image: my-worker:latest
+   logging:
+     driver: "json-file"
+     options:
+       max-size: "50m"
+       max-file: "7"
+       compress: "true"

For high-volume production workloads, migrate off json-file entirely:

- "log-driver": "json-file"
+ "log-driver": "local"

The local driver uses protobuf binary format with better rotation handling and is the Docker-recommended replacement. docker logs still works. Filebeat/Fluentd need the docker input plugin, not raw file tailing.

---

💡 Tired of pasting proprietary configs into ChatGPT? Generic AI tools log your company's ARNs, DB strings, and private keys. StackEngine is a zero-backend, pure Client-Side WASM utility. Drop your failing config into the sandbox above. We redact your secrets locally in the browser and auto-generate the refactored code using your own API key.

---

Prevention in CI/CD

1. Lint daemon.json in Your Ansible/Terraform Pipelines

# Use jq to assert max-file is set and >= 5
cat /etc/docker/daemon.json | jq -e '.["log-opts"]["max-file"] | tonumber >= 5' || exit 1

2. Checkov Policy for Compose Files

Checkov rule CKV_DOCKER_7 checks for log driver configuration. Add to your pre-commit or CI stage:

checkov -f docker-compose.yml --check CKV_DOCKER_7

Write a custom check if you need to enforce max-file minimums:

# checkov custom check skeleton
from checkov.common.models.enums import CheckResult
from checkov.dockerfile.checks.base_dockerfile_check import BaseDockerfileCheck

class LogRotationCheck(BaseDockerfileCheck):
    def __init__(self):
        super().__init__(name="Ensure json-file max-file >= 5", id="CKV_CUSTOM_LOG_001", ...)

3. OPA/Conftest Policy for Docker Compose

# policy/log_rotation.rego
package docker.compose

deny[msg] {
  service := input.services[name]
  not service.logging.options["max-file"]
  msg := sprintf("Service '%v' missing log max-file rotation config", [name])
}

deny[msg] {
  service := input.services[name]
  to_number(service.logging.options["max-file"]) < 5
  msg := sprintf("Service '%v' max-file must be >= 5", [name])
}

conftest test docker-compose.yml --policy policy/

4. Monitoring — Alert on 0-Byte Log Files

# Cron or Prometheus node-exporter textfile collector
find /var/lib/docker/containers -name '*-json.log' -size 0 | wc -l

If this returns > 0 on a running system with active containers, you have broken log rotation. Page on it.