Terraform S3 DynamoDB Lock ID Mismatch

The Incident (What Does the Error Mean?)

Raw error output from a failed terraform apply or terraform plan:

Error: Failed to lock state

Error acquiring the state lock: ConditionalCheckFailedException: The conditional request failed

Lock Info:
  ID:        a3f1c2d4-7e89-4b12-a1d0-5f6e7c8b9012
  Path:      s3://my-tf-state-bucket/prod/terraform.tfstate
  Operation: OperationTypePlan
  Who:       ci-runner@gitlab-runner-prod
  Version:   1.6.3
  Created:   2024-11-14 03:22:11.408742 +0000 UTC
  Info:

Terraform acquires a state lock to protect the state from being written
by multiple users at the same time. Please resolve the issue above and
try again. For most commands, you can disable locking with the
"-lock=false" flag, but this is not recommended.

Immediate consequence: Every subsequent plan, apply, or destroy is blocked. Your CI/CD pipeline is dead. If this is a production environment, no infrastructure changes can be safely deployed until the lock is cleared. The DynamoDB ConditionalCheckFailedException means Terraform's PutItem with a condition expression (attribute_not_exists(LockID)) failed — a lock record already exists for this state path.

The Attack Vector / Blast Radius

This is not just an inconvenience — understand the full failure surface:

Scenario 1 — Stale lock from a crashed CI runner (most common): A GitLab/GitHub Actions runner was killed mid-apply (OOM kill, spot instance termination, pipeline timeout). Terraform never ran its deferred unlock. The DynamoDB item persists indefinitely. Every subsequent pipeline run hits ConditionalCheckFailedException.

Scenario 2 — Backend path drift: Someone changed the key parameter in the S3 backend config (e.g., renamed workspace path or environment prefix). Terraform now generates a different LockID composite key (<bucket>/<key>) but the old lock item in DynamoDB still holds the previous path. The table entry and the new lock request are for different logical paths — Terraform's lock ID validation fails.

Scenario 3 — Concurrent apply from two runners: Two CI pipelines triggered simultaneously (e.g., a merge and a manual run). One acquired the lock legitimately. The second is now blocked. This is the correct behavior — but if the first runner is stuck or dead, you still end up with an orphaned lock.

Blast radius if you use -lock=false carelessly:

• Two concurrent apply operations write to the same .tfstate simultaneously → state file corruption.
• Corrupted state = Terraform loses track of real infrastructure → duplicate resources, failed destroys, billing blowout.
• In a worst case with remote state and no versioning on the S3 bucket, the corrupted state is unrecoverable without manual reconciliation.

DynamoDB table structure — what you're actually fighting:

The lock item stored in DynamoDB looks like this:

{
  "LockID": { "S": "my-tf-state-bucket/prod/terraform.tfstate" },
  "Info": {
    "S": "{\"ID\":\"a3f1c2d4-7e89-4b12-a1d0-5f6e7c8b9012\",\"Operation\":\"OperationTypePlan\",\"Who\":\"ci-runner@gitlab-runner-prod\",\"Version\":\"1.6.3\",\"Created\":\"2024-11-14T03:22:11.408742Z\"}"
  }
}

The LockID attribute is the partition key of the DynamoDB table. Terraform uses a conditional write (attribute_not_exists(LockID)) to acquire it atomically. A mismatch or stale entry breaks this atomic check.

Prevention in CI/CD

1. Always wrap Terraform in a lock-aware CI pattern:

# .gitlab-ci.yml — safe Terraform apply pattern
terraform-apply:
  script:
    - terraform init
    - terraform plan -out=tfplan
    - terraform apply tfplan
  after_script:
    # Force-unlock on job cancellation or failure using stored lock ID
    - |
      LOCK_ID=$(cat .terraform/lock-id 2>/dev/null || echo "")
      if [ -n "$LOCK_ID" ]; then
        terraform force-unlock -force $LOCK_ID || true
      fi
  interruptible: false  # CRITICAL: prevents mid-apply cancellation

2. Enable S3 bucket versioning — non-negotiable:

 resource "aws_s3_bucket_versioning" "tf_state" {
   bucket = aws_s3_bucket.tf_state.id
   versioning_configuration {
-    status = "Disabled"
+    status = "Enabled"
   }
 }

Without versioning, a corrupted state write is permanent. With versioning, you can roll back to the last known-good state.

3. Checkov policy to enforce DynamoDB locking is configured:

checkov -d . --check CKV_TF_3  # Checks: ensure S3 backend uses DynamoDB for locking

4. OPA/Sentinel policy — enforce lock table is always set:

# opa/terraform_backend_lock.rego
package terraform.backend

deny[msg] {
  backend := input.configuration.backend_config
  backend.type == "s3"
  not backend.config.dynamodb_table
  msg := "S3 backend MUST specify a DynamoDB lock table. No exceptions."
}

5. Monitor for stale locks with a CloudWatch alarm:

Set up a scheduled Lambda that scans the DynamoDB lock table for items older than 2 hours and fires a CloudWatch alarm or PagerDuty alert. A lock older than your maximum apply window is definitionally orphaned and should auto-page on-call.