Docker Swarm Service Update Sync Tool

The Incident (What Does the Error Mean?)

Raw error output from the Docker daemon or CLI:

Error response from daemon: rpc error: code = Unknown
desc = update out of sync

Or during a docker stack deploy:

failed to update service mystack_api: Error response from daemon: update out of sync

Immediate consequence: The service updater on the Raft-leader manager node has detected that the version of the service spec it holds does not match what was last committed to the Raft log, or that in-flight task slots are in a state the reconciler cannot advance. The rolling update is completely halted. No new tasks are scheduled. Existing tasks may be in Running, Failed, or Shutdown states simultaneously — your service is partially updated and in an undefined state.

The Attack Vector / Blast Radius

This is a distributed state consistency failure, not a simple CLI error. The blast radius:

• Partial rollout lock: Some replicas run the new image, some run the old. Load balancer (the Swarm ingress mesh) routes traffic to both. If the new image has a breaking schema change, you now have split-brain application behavior hitting a single VIP.
• Cascading update block: Any subsequent docker service update on the same service will also fail until the stuck state is cleared. Automated CD pipelines (Jenkins, GitLab CI, ArgoCD with Swarm plugins) will loop and retry, potentially hammering the manager API and degrading Raft performance for the entire cluster.
• Manager leader instability: If the desync was caused by a manager failover mid-update (the most common root cause), the new leader may have an incomplete Raft log entry for the update transaction. Forcing another update before clearing this can corrupt the service spec in the Raft store — requiring a full service delete and recreate.
• Quorum risk: On a 3-manager cluster, if one manager is already degraded (causing the original failover), you are now one manager failure away from losing quorum and freezing the entire cluster's control plane.

Root causes ranked by frequency:

1. Manager leader election occurred during an active service update transaction
2. Concurrent docker service update calls on the same service (CI pipeline race condition)
3. Worker node eviction while tasks were being rescheduled mid-update
4. Clock skew between manager nodes causing Raft log version conflicts
5. Corrupted service spec from a partial docker stack deploy

Prevention in CI/CD

1. Serialize service updates in your pipeline

Never allow concurrent docker service update calls on the same service. In GitLab CI or GitHub Actions:

# GitHub Actions — use concurrency groups to serialize Swarm deployments
concurrency:
  group: swarm-deploy-${{ github.ref }}
  cancel-in-progress: false

2. Poll for convergence before proceeding

Do not fire-and-forget docker service update. Block the pipeline until convergence:

#!/bin/bash
# wait-for-service-stable.sh
SERVICE=$1
TIMEOUT=120
ELAPSED=0

until [ "$(docker service inspect $SERVICE --format '{{.UpdateStatus.State}}')" = "completed" ]; do
  if [ $ELAPSED -ge $TIMEOUT ]; then
    echo "ERROR: Service update did not converge in ${TIMEOUT}s"
    docker service rollback $SERVICE
    exit 1
  fi
  sleep 5
  ELAPSED=$((ELAPSED+5))
done

3. Validate Compose files with `docker-compose config` in CI

# Pre-deploy lint step
- name: Validate stack config
  run: |
    docker-compose -f docker-compose.prod.yml config --quiet
    # Fail fast on spec errors before touching the live cluster

4. OPA/Conftest policy to enforce safe update_config

# policy/swarm_update_config.rego
package swarm

deny[msg] {
  svc := input.services[_]
  svc.deploy.update_config.failure_action == "continue"
  msg := sprintf("Service '%v': failure_action must not be 'continue' — use 'rollback' or 'pause'", [svc.name])
}

deny[msg] {
  svc := input.services[_]
  svc.deploy.update_config.parallelism > 2
  svc.deploy.replicas <= 4
  msg := sprintf("Service '%v': parallelism too high relative to replica count — risk of full-service desync", [svc.name])
}

Run in CI with conftest test docker-compose.prod.yml --policy policy/.

5. Monitor manager health proactively

# Alert if manager count drops below quorum
docker node ls --filter role=manager --format '{{.Status}}' | grep -c Ready
# If < 2 on a 3-manager cluster, page immediately — next service update is a desync risk