Windows Hyper-V Docker Inspect Hang Fix

The Incident (What Does the Error Mean?)

You run:

docker inspect 3f9a1c2b8d04

The shell hangs. No output. No timeout. Ctrl+C returns you to the prompt but the daemon is still blocked internally. Meanwhile:

docker ps       # also hangs
docker stop     # also hangs
docker rm -f    # also hangs

Checking the Docker daemon log at C:\ProgramData\Docker\panic.log or via Get-EventLog reveals:

time="2024-11-12T03:17:42Z" level=error msg="hcsshim::Container Wait" 
error="the handle is invalid." id="3f9a1c2b8d04"

time="2024-11-12T03:17:42Z" level=warning msg="failed to wait for container" 
container=3f9a1c2b8d04 error="hcsshim: the handle is invalid"

ERROR: context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Immediate consequence: The Docker daemon's goroutine pool is blocked on a synchronous HCS named pipe call that will never return. Every subsequent docker CLI command that touches container state queues behind this deadlock. Your entire Docker host is effectively frozen for container operations.

The Attack Vector / Blast Radius

This is not a security exploit — it is a cascading availability failure with the following blast radius:

Why HCS deadlocks: Each Hyper-V isolated container runs inside a lightweight VM. The Docker daemon communicates with these VMs via hcsshim over a Windows named pipe (\\.\pipe\docker_engine). When the VM worker process (vmwp.exe) crashes mid-operation — during a snapshot, a checkpoint, or a network namespace teardown — hcsshim holds an open handle to a now-invalid VM compute system. The daemon's container.Wait() call blocks on that handle forever. There is no default timeout in older Docker Engine versions on Windows.

Cascading failure chain:

1. One orphaned vmwp.exe → daemon goroutine blocked
2. Daemon goroutine pool exhaustion → ALL docker CLI commands time out
3. Orchestrator (Kubernetes kubelet, Swarm agent) health checks fail → node marked NotReady
4. Workloads rescheduled to other nodes → potential overload on remaining nodes
5. If this is a CI/CD runner node, all pipelines queue indefinitely

In Kubernetes on Windows nodes, this manifests as the kubelet unable to garbage-collect terminated pods, leading to disk pressure from unremoved container layers on the C:\ volume.

Prevention in CI/CD

1. Watchdog Script on Windows Nodes

Deploy a scheduled task that detects and recovers from HCS deadlocks:

# watchdog.ps1 — run every 2 minutes via Task Scheduler
$timeout = 10  # seconds
$job = Start-Job { docker ps --no-trunc }
if (-not (Wait-Job $job -Timeout $timeout)) {
    Remove-Job $job -Force
    Write-EventLog -LogName Application -Source "DockerWatchdog" -EntryType Warning -EventId 1001 -Message "Docker daemon hung. Recycling HCS."
    Stop-Service docker -Force
    Stop-Service vmcompute -Force
    Start-Service vmcompute
    Start-Service docker
}

2. Kubernetes Node Problem Detector

If running Windows nodes in AKS or self-managed Kubernetes, deploy node-problem-detector with a custom rule:

# npd-config.json rule
{
  "type": "permanent",
  "condition": "DockerHung",
  "reason": "HCSDeadlock",
  "message": "docker inspect hang detected — HCS named pipe unresponsive",
  "pattern": "hcsshim.*handle is invalid"
}

This triggers a node taint automatically, draining workloads before the node fully freezes.

3. CI/CD Pipeline Gate (GitHub Actions / Azure DevOps)

Add a pre-flight check before any Windows container job:

- name: Verify Docker daemon responsive
  shell: pwsh
  run: |
    $job = Start-Job { docker version }
    if (-not (Wait-Job $job -Timeout 15)) {
      Remove-Job $job -Force
      Write-Error "Docker daemon unresponsive on Windows runner. Failing fast."
      exit 1
    }
    Receive-Job $job

4. Upgrade Path

• Docker Engine 24.0+ on Windows includes improved HCS handle cleanup on daemon restart.
• containerd 1.7+ as the Windows runtime (replacing the legacy docker-containerd) has significantly better Hyper-V VM lifecycle management and does not exhibit this named pipe deadlock pattern.
• Migrate to containerd + nerdctl on Windows nodes where possible — this eliminates the hcsshim v1 code path entirely.