RAPID INCIDENT TRIAGE & ACTION PLAN•Security
01. Error Signature
Terraform IAM Role Assume Apply Auditor
02. Root Cause Vector

Terraform apply fails with 'Error: failed to assume role' when the IAM trust policy, STS permissions, or external ID configuration is mis...

03. Rapid Prescription

Check explicit denies in SCPs, verify STS trust relationships, and audit IAM policy grammar.

Zero-Trust Client IDE/ Security

Terraform IAM Role Assume Apply Auditor

Terraform apply fails with 'Error: failed to assume role' when the IAM trust policy, STS permissions, or external ID configuration is misconfigured, blocking all infrastructure deployments.

SYSTEM STATUS
CREDITS: 0 / 3
source.yaml(43L)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
#!/usr/bin/env bash
# TODO: This script is a mess but it works in prod? Needs refactoring after PCI audit.
# Legacy deployment script for pci-secure-zone cluster (DO NOT MODIFY WITHOUT APPROVAL)

# Load env from old config manager (deprecated)
source /opt/legacy/env-loader.sh 2>/dev/null || true

# AWS credentials for terraform backend - rotated quarterly by security team
export AWS_ACCESS_KEY_ID="AKIAIOSFODNN7EXAMPLE"
export AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

# Stripe API key for billing microservice (TODO: move to vault)
STRIPE_LIVE_KEY="sk_live_4eC39HqLyjWDarjtT1zdp7dc"

# Database connection URI for legacy reporting
DB_URI="postgresql://app_user:SuperSecret123!@pci-db.internal:5432/legacy_reports?sslmode=require"

# Assume role ARN from central IAM (hardcoded for now due to DNS issues in vpc-0a1b2c3d)
ASSUME_ROLE_ARN="arn:aws:iam::123456789012:role/terraform-prod-apply"

# Override with regional role for DR failover (commented out after last incident)
# ASSUME_ROLE_ARN="arn:aws:iam::210987654321:role/terraform-dr-apply"

# VULNERABILITY: The role ARN is valid, but the trust policy only allows assumption from specific VPCs not yet configured
# This script runs from a jump host outside the allowed network scope.

# SSH key for bastion access (RSA 2048)
BASTION_KEY="-----BEGIN RSA PRIVATE KEY-----\nMIIEowIBAAKCAQEArL5kY6H6z8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qF\nz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8q\nFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8q\nFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8q\nFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8qFz8q\n-----END RSA PRIVATE KEY-----"

# Run terraform with assumed role (using old wrapper due to MFA requirements)
# TODO: Replace with aws-vault once IAM team approves
terraform init -backend-config="role_arn=${ASSUME_ROLE_ARN}" -reconfigure

# Apply with explicit role assumption (should fail due to missing source identity condition)
terraform apply -auto-approve \
  -var="assume_role_arn=${ASSUME_ROLE_ARN}" \
  -var="region=us-east-1" \
  -var-file="environments/prod.tfvars"

# Fallback to local profile if assume fails (never tested)
# terraform apply -auto-approve -var="use_local_creds=true"

echo "Deployment complete (or failed silently, check logs)."
AI Audit Console
> Waiting for pre-flight audit...
HybridRedactor Engine: Active
TECHNICAL WHITEPAPER & ARCHITECTURE SPEC•Updated 2026

How to Fix Terraform 'Error: failed to assume role' IAM – Root Cause & Complete Solution

Threat/Impact Level: CRITICAL | Exploitability/Downtime Risk: HIGH | Time to Fix: 10–20 mins

TL;DR

  • What broke: Terraform's AWS provider cannot call sts:AssumeRole — the trust policy on the target role doesn't authorize the caller, the external_id is missing or wrong, or the source identity lacks sts:AssumeRole in its permission policy.
  • How to fix it: Audit the trust relationship on the target IAM role, verify the external_id matches exactly, and confirm the calling principal has an explicit sts:AssumeRole allow in its attached policy.
  • Shortcut: Use our Client-Side Sandbox above to auto-refactor this — paste your provider block and the role's trust policy JSON and get a corrected diff instantly.

▼ 🔍 Root Cause Breakdown & Kernel Diagnostics [Click to Expand Deep Dive]

The Incident (What Does the Error Mean?)

Raw error output from terraform apply:

Error: failed to assume role

  with provider["registry.terraform.io/hashicorp/aws"],
  on main.tf line 3, in provider "aws":
   3:   assume_role {

AWS Error: AccessDenied: User: arn:aws:iam::123456789012:role/ci-runner
is not authorized to assume role: arn:aws:iam::987654321098:role/terraform-deploy-role
        status code: 403, request id: a1b2c3d4-...

Immediate consequence: Every single terraform plan and terraform apply in this workspace is dead. No resources can be created, updated, or destroyed. If this is your CI/CD pipeline, every deployment is blocked. If this is a cross-account setup, all cross-account automation is offline.

This is not a transient error. It will not self-heal. STS returned a hard 403 AccessDenied — the assume-role handshake failed at the IAM control plane level.


▼ 🔍 Detailed Impact & Blast Radius Analysis [Click to Expand Deep Dive]

The Attack Vector / Blast Radius

This error surfaces in three distinct failure scenarios, each with its own blast radius:

Scenario 1 — Trust Policy Gap (Most Common) The target role's trust policy (sts:AssumeRole principal) does not list the calling entity. If you recently rotated the CI runner's IAM role ARN, renamed an instance profile, or migrated from IAM users to IAM roles, the trust relationship is stale. Every cross-account and same-account automation using this role is broken.

Scenario 2 — Missing external_id (Third-Party / Multi-Tenant Risk) If the role was created with a required external_id condition (standard for vendor integrations and multi-tenant SaaS), omitting it in the Terraform provider block causes a silent 403. Worse: if external_id is present in the trust policy but your Terraform config doesn't send it, an attacker who gains access to your caller ARN cannot assume the role — so this is a security control, not just a config nuisance. Breaking it means your automation is locked out, but fixing it carelessly (removing the condition) opens a confused deputy attack surface.

Scenario 3 — Permission Boundary or SCP Blocking sts:AssumeRole AWS Organizations Service Control Policies (SCPs) or IAM permission boundaries on the calling role can explicitly deny sts:AssumeRole even if the trust policy is correct. This is the hardest to debug because the trust policy looks fine. An SCP Deny is an explicit override that cannot be overridden by any identity policy.


How to Fix It (The Solution)

Step 1 — Identify the Exact Failure Point

Run this before touching any Terraform:

# Test assume-role manually with the exact ARN Terraform uses
aws sts assume-role \
  --role-arn "arn:aws:iam::987654321098:role/terraform-deploy-role" \
  --role-session-name "debug-session" \
  --external-id "your-external-id-if-required"

# Check the calling identity
aws sts get-caller-identity

If this returns AccessDenied, the problem is confirmed at the STS layer — not Terraform.


Basic Fix — Correct the Trust Policy

Navigate to IAM → Roles → terraform-deploy-role → Trust relationships and verify the principal.

# IAM Trust Policy JSON on the TARGET role (terraform-deploy-role)
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
-       "AWS": "arn:aws:iam::123456789012:role/old-ci-runner"
+       "AWS": "arn:aws:iam::123456789012:role/ci-runner"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
-         "sts:ExternalId": "wrong-external-id"
+         "sts:ExternalId": "correct-external-id-matching-provider-block"
        }
      }
    }
  ]
}

Basic Fix — Correct the Terraform Provider Block

# provider.tf
provider "aws" {
  region = "us-east-1"

  assume_role {
-   role_arn     = "arn:aws:iam::987654321098:role/wrong-role-name"
+   role_arn     = "arn:aws:iam::987654321098:role/terraform-deploy-role"
+   session_name = "terraform-ci-session"
+   external_id  = "correct-external-id-matching-trust-policy"
  }
}

Enterprise Best Practice — Least-Privilege Cross-Account Role with Conditions

Never use a bare principal without condition keys. Lock down the trust policy with aws:PrincipalArn, sts:ExternalId, and optionally aws:MultiFactorAuthPresent for human roles.

# Terraform resource: the TARGET role's trust policy (in the target account)
resource "aws_iam_role" "terraform_deploy_role" {
  name = "terraform-deploy-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect    = "Allow"
        Principal = {
-         AWS = "arn:aws:iam::123456789012:root"
+         AWS = "arn:aws:iam::123456789012:role/ci-runner"
        }
        Action = "sts:AssumeRole"
        Condition = {
          StringEquals = {
+           "sts:ExternalId"      = var.external_id
+           "aws:PrincipalArn"    = "arn:aws:iam::123456789012:role/ci-runner"
          }
+         Bool = {
+           "aws:SecureTransport" = "true"
+         }
        }
      }
    ]
  })
}

# Terraform resource: permission policy on the CALLING role (in the source account)
resource "aws_iam_role_policy" "allow_assume_deploy_role" {
  name = "allow-assume-terraform-deploy-role"
  role = aws_iam_role.ci_runner.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
-       Action   = "sts:*"
-       Resource = "*"
+       Action   = "sts:AssumeRole"
+       Resource = "arn:aws:iam::987654321098:role/terraform-deploy-role"
      }
    ]
  })
}

Key hardening decisions:

  • sts:* on * is a critical misconfiguration — scope to exact action and exact role ARN.
  • Using arn:aws:iam::ACCOUNT:root as principal allows any identity in that account to assume the role if they have sts:AssumeRole — always pin to the specific role ARN.
  • external_id stored in var.external_id should be sourced from a secrets manager, not hardcoded.

💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.

(For CI/CD pipeline automation, install via npm i -g stackengine).


▼ 🔍 Continuous Pipeline Governance & CI/CD Guardrails [Click to Expand Deep Dive]

Prevention in CI/CD

1 — Checkov Static Analysis (Catches Wildcard STS Before Apply)

Add to your CI pipeline:

# .github/workflows/terraform.yml
- name: Run Checkov IAM Scan
  uses: bridgecrewio/checkov-action@master
  with:
    directory: .
    check: CKV_AWS_110,CKV_AWS_111,CKV_AWS_49
    framework: terraform

CKV_AWS_49 specifically flags IAM roles with overly permissive trust policies.

2 — OPA / Conftest Policy (Enforce `external_id` Requirement)

# policies/iam_assume_role.rego
package terraform.iam

deny[msg] {
  resource := input.resource_changes[_]
  resource.type == "aws_iam_role"
  policy := json.unmarshal(resource.change.after.assume_role_policy)
  stmt := policy.Statement[_]
  stmt.Effect == "Allow"
  stmt.Action == "sts:AssumeRole"
  not stmt.Condition.StringEquals["sts:ExternalId"]
  msg := sprintf("Role '%v' trust policy allows sts:AssumeRole without sts:ExternalId condition — confused deputy risk.", [resource.name])
}

Run in CI:

terraform show -json tfplan.binary | conftest test --policy policies/ -

3 — `terraform validate` + Provider Auth Pre-Flight

Add an explicit STS identity check as the first CI step — fail fast before terraform plan wastes time:

# ci-preflight.sh
set -e
echo "Verifying caller identity..."
aws sts get-caller-identity
echo "Testing role assumption..."
aws sts assume-role \
  --role-arn "${TF_VAR_deploy_role_arn}" \
  --role-session-name "ci-preflight" \
  --external-id "${TF_VAR_external_id}" \
  --query 'AssumedRoleUser.Arn' \
  --output text
echo "Role assumption verified. Proceeding with terraform plan."

4 — AWS Config Rule for Trust Policy Drift

Deploy AWS_IAM_ROLE_TRUST_POLICY_CHANGE Config rule with SNS alerting. Any modification to a trust relationship triggers an alert to your security channel — catches accidental or malicious trust policy changes before the next deployment fails.

Official Grounding & Normative Specifications

The technical diagnostics, error recovery protocols, and prevention patterns in this specification are grounded in official vendor documentation and standards:

Diagnostic & Architectural FAQ

Q1.Why does 'terraform apply' fail with 'failed to assume role' even though the IAM role exists?
The root cause is a broken trust handshake: the calling identity must be explicitly listed as a trusted principal in the target role's trust policy AND have sts:AssumeRole permissions. A single misconfigured ARN or missing condition breaks the entire pipeline. This is a fundamental IAM authority flaw that manual debugging with 'aws sts assume-role' only confirms—it doesn't fix. Drop the broken manifest into the StackEngine Local Sandbox above to automatically compile a zero-trust fix with correct principal and condition bindings, eliminating guesswork and credential exposure.
Q2.What is 'external_id' in Terraform's assume_role block and do I always need it?
The external_id is a critical security token that prevents confused deputy attacks by ensuring only authorized entities can assume the role. Omitting it when the trust policy requires it causes a silent 403, creating a severe security vulnerability and deployment failure. Manually synchronizing these secrets across accounts and vendors is error-prone and risks secret sprawl. Drop the broken manifest into the StackEngine Local Sandbox above to automatically compile a zero-trust fix with validated external_id enforcement, preventing lateral movement and compliance violations.
Q3.How do I debug 'failed to assume role' when the trust policy looks correct?
When trust policies appear correct, the failure is often in hidden enforcement layers: AWS Organizations SCPs, IAM Permission Boundaries, or MaxSessionDuration mismatches. Manually auditing these across accounts is a compliance nightmare and scales poorly, leading to toxic drift and audit failures. For large mono-repos or enterprise-scale deployments, upgrade to the StackEngine CI/CD Automator (Team Edition) for on-prem, automated pull request generation with Maker-Checker review, policy synthesis, and continuous compliance guardrails across your entire infrastructure.