Terraform S3 Backend State Lock Resolver

The Incident (What Does the Error Mean?)

Raw error output:

Error: Error locking state: Error acquiring the state lock: ConditionalCheckFailedException: The conditional request failed
	state file: s3://my-tf-state-bucket/prod/terraform.tfstate
	Lock Info:
	  ID:        a1b2c3d4-e5f6-7890-abcd-ef1234567890
	  Path:      s3://my-tf-state-bucket/prod/terraform.tfstate
	  Operation: OperationTypeApply
	  Who:       runner@ci-node-07
	  Version:   1.5.7
	  Created:   2024-01-15 03:42:11.823456789 +0000 UTC
	  Info:

Terraform acquires a state lock to protect the state from being written
by multiple users at the same time. Please resolve the issue above and try
again. For most commands, you can disable locking with the "-lock=false"
flag, but this is not recommended.

Immediate consequence: Every terraform plan, apply, and destroy is dead until the lock is cleared. Your pipeline is halted. If a team member bypasses this with -lock=false, you risk concurrent state writes and state corruption — which is significantly worse than the lock itself.

The Attack Vector / Blast Radius

This is not just an inconvenience. The blast radius has two failure modes:

1. Operational: Stale lock from a dead process CI runners get OOM-killed. Engineers hit Ctrl+C mid-apply. GitHub Actions jobs time out. In every case, Terraform never gets to release the lock. The DynamoDB LockTable retains the LockID record indefinitely. Your entire team is now blocked on a ghost process.

2. Security: The -lock=false temptation Under pressure during an outage, engineers reach for terraform apply -lock=false. This disables the only concurrency control protecting your state file. If two engineers or two CI jobs run simultaneously with locking disabled:

• State writes race and one apply silently overwrites the other
• Resources get orphaned — created in AWS but absent from state
• Next terraform apply sees drift and destroys live infrastructure to reconcile

If your S3 bucket is misconfigured (no versioning, no MFA delete): a corrupted state write is unrecoverable. You are now manually reconstructing state for a production environment.

Prevention in CI/CD

1. Always wrap Terraform in a lock-aware CI wrapper

#!/bin/bash
# ci-terraform-apply.sh
set -euo pipefail

cleanup() {
  echo "Runner exiting — releasing lock if held"
  terraform force-unlock -force "${LOCK_ID:-}" 2>/dev/null || true
}
trap cleanup EXIT SIGTERM SIGINT

terraform apply -auto-approve

2. Checkov: Enforce S3 versioning and encryption on state buckets

# .checkov.yaml
checks:
  - CKV_AWS_77   # S3 versioning enabled
  - CKV_AWS_119  # DynamoDB encrypted at rest
  - CKV_AWS_28   # DynamoDB point-in-time recovery

Run in your pipeline:

checkov -d . --check CKV_AWS_77,CKV_AWS_119,CKV_AWS_28 --hard-fail-on HIGH

3. OPA Policy: Block -lock=false in Atlantis/Terraform Cloud

# policy/no_lock_disable.rego
package terraform.deny_lock_false

deny[msg] {
  input.flags[_] == "-lock=false"
  msg := "POLICY VIOLATION: -lock=false is prohibited in automated pipelines. Resolve the stale lock manually."
}

4. CloudWatch Alarm on stale locks

# Alert if any DynamoDB lock record is older than 30 minutes
aws cloudwatch put-metric-alarm \
  --alarm-name "TerraformStaleLock" \
  --metric-name "OldestUnprocessedAge" \
  --namespace "TerraformLocks" \
  --threshold 1800 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 1 \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:ops-alerts

Pair this with a Lambda that queries DynamoDB for lock records with Created timestamps older than your longest expected apply window and pages on-call automatically.