Docker Remote Daemon CP Unexpected EOF

Threat/Impact Level: HIGH | Downtime Risk: HIGH | Time to Fix: 15–45 mins

TL;DR

• What broke: docker cp opens a raw HTTP/TCP stream to the remote daemon; any intermediary (nginx proxy, SSH timeout, NAT gateway, low ulimit) that closes the connection mid-transfer kills the tar stream with unexpected EOF.
• How to fix it: Switch transport to SSH (DOCKER_HOST=ssh://user@host), tune proxy/daemon timeouts, or stream via docker exec + tar pipe instead of docker cp.
• Use our Client-Side Sandbox below to paste your docker cp invocation and daemon.json — it will auto-refactor the transport and timeout config without sending your host addresses anywhere.

docker -H tcp://10.0.1.55:2376 cp ./backup-20GB.tar.gz mycontainer:/data/

Error response from daemon: unexpected EOF
# or
error during connect: Post "http://10.0.1.55:2376/v1.41/containers/mycontainer/archive?path=%2Fdata%2F": EOF

How to Fix It

Root Cause Checklist (run these first)

# 1. Check daemon reachability and version
docker -H tcp://10.0.1.55:2376 info 2>&1 | grep -E 'Server Version|Operating System|Total Memory'

# 2. Check available memory on daemon host — tar buffers large files in memory
ssh user@10.0.1.55 'free -h && df -h /var/lib/docker'

# 3. Check ulimits on the daemon process
ssh user@10.0.1.55 'cat /proc/$(pgrep dockerd)/limits | grep -E "open files|file size"'

Basic Fix 1 — Switch to SSH Transport (Eliminates Proxy Timeout Layer)

- docker -H tcp://10.0.1.55:2376 cp ./backup-20GB.tar.gz mycontainer:/data/
+ DOCKER_HOST=ssh://deploy@10.0.1.55 docker cp ./backup-20GB.tar.gz mycontainer:/data/

SSH transport uses the system SSH connection (with keepalives) instead of a raw TCP stream through whatever proxy sits in front of port 2376. This alone fixes 70% of EOF cases.

Enable SSH keepalives to prevent idle disconnection on long transfers:

# ~/.ssh/config on the CLIENT machine
+ Host 10.0.1.55
+   ServerAliveInterval 15
+   ServerAliveCountMax 8
+   Compression no

Compression no is intentional — SSH compression on already-compressed archives wastes CPU and can cause buffer stalls on large streams.

Basic Fix 2 — Stream via tar pipe instead of docker cp

When docker cp is unreliable, bypass it entirely:

- docker -H tcp://10.0.1.55:2376 cp ./backup-20GB.tar.gz mycontainer:/data/
+ cat ./backup-20GB.tar.gz | \
+   DOCKER_HOST=ssh://deploy@10.0.1.55 \
+   docker exec -i mycontainer bash -c 'cat > /data/backup-20GB.tar.gz'

For directories, use tar-over-exec which is what docker cp does internally, but you control the stream:

tar czf - ./mydir | DOCKER_HOST=ssh://deploy@10.0.1.55 docker exec -i mycontainer tar xzf - -C /data/

Enterprise Best Practice — Tune daemon.json and Proxy Timeouts

If you must use TCP transport behind nginx:

# /etc/nginx/nginx.conf (TCP stream proxy block)
stream {
  server {
    listen 2376;
    proxy_pass 10.0.1.55:2376;
-   # no timeout configured — defaults to 60s idle
+   proxy_timeout 3600s;
+   proxy_connect_timeout 10s;
  }
}

# /etc/docker/daemon.json on the remote host
{
+ "max-concurrent-downloads": 3,
+ "max-concurrent-uploads": 3,
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  }
}

For AWS NLB: set idle timeout to 3600 seconds in the Target Group attributes. The default 350s will kill any transfer of a file >~500MB on a 10Mbps link.

💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.

(For CI/CD pipeline automation, install via npm i -g stackengine).

# .gitlab-ci.yml / GitHub Actions step
- docker cp ./artifact.tar mycontainer:/app/
+ docker cp ./artifact.tar mycontainer:/app/ || { echo "docker cp failed — aborting"; exit 1; }

# Generate checksum locally
sha256sum ./artifact.tar > artifact.tar.sha256

# Copy both
DOCKER_HOST=ssh://deploy@host docker cp ./artifact.tar mycontainer:/app/
DOCKER_HOST=ssh://deploy@host docker cp ./artifact.tar.sha256 mycontainer:/app/

# Verify inside container
DOCKER_HOST=ssh://deploy@host docker exec mycontainer bash -c \
  'cd /app && sha256sum -c artifact.tar.sha256 || exit 1'

# policy/docker_transport.rego
package docker.transport

deny[msg] {
  input.env.DOCKER_HOST
  startswith(input.env.DOCKER_HOST, "tcp://")
  msg := "DOCKER_HOST must use ssh:// transport. Raw TCP is prohibited in CI pipelines."
}