RAPID INCIDENT TRIAGE & ACTION PLAN•Security
01. Error Signature
AWS STS GetCallerIdentity Token Verifier
02. Root Cause Vector

AWS STS GetCallerIdentity rejects temporary credentials with an invalid security token error, halting identity verification and breaking ...

03. Rapid Prescription

Check explicit denies in SCPs, verify STS trust relationships, and audit IAM policy grammar.

Zero-Trust Client IDE/ Security

AWS STS GetCallerIdentity Token Verifier

AWS STS GetCallerIdentity rejects temporary credentials with an invalid security token error, halting identity verification and breaking downstream IAM-dependent workflows.

SYSTEM STATUS
CREDITS: 0 / 3
source.yaml(69L)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
#!/usr/bin/env python3
# TODO: This script is a mess, but it's been working for 3 years. Don't touch unless you have to.
# Legacy system for PCI-secure-zone compliance checks. Uses old boto3 version.

import boto3
import os
import sys
from datetime import datetime, timedelta

# ========== CONFIGURATION BLOCK (DO NOT MODIFY WITHOUT APPROVAL) ==========
# TODO: Migrate to Secrets Manager (JIRA ticket: SEC-4456, status: backlog)
AWS_ACCESS_KEY_ID = "AKIAIOSFODNN7EXAMPLE"
AWS_SECRET_ACCESS_KEY = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
# Old key from previous rotation, might still be cached somewhere
# AWS_SECRET_ACCESS_KEY = "xJblrYVtnGEMJ/L8NDFOG/cQySgjDZEXAMPLEKEY"

# Stripe key for billing webhook validation (PCI-DSS scope)
STRIPE_API_KEY = "sk_live_4eC39HqLyjWDarjtT1zdp7dc"

# Database connection for audit logs (internal only)
DB_URI = "postgresql://audit_user:SuperSecretPass123!@pci-db.internal.legacy-api:5432/audit_db"

# Temporary session token from assumed role (gets overridden later)
AWS_SESSION_TOKEN = "FQoGZXIvYXdzE///////////wEaDLwk7m8EXAMPLE_TOKEN"
# ========== END CONFIGURATION ==========

# Function to verify caller identity (used by compliance scanner)
def verify_caller_identity():
    """
    Uses AWS STS GetCallerIdentity to validate temporary credentials.
    This is called by the nightly PCI compliance job.
    """
    try:
        # NOTE: We manually set the token here because the env var is unreliable after the Lambda cold start hack.
        # VULNERABILITY: The session token is explicitly set to an empty string after being loaded from config,
        # but the credentials object is created before that, using the token from the environment/global scope.
        # This creates a race/state conflict where the token is invalidated between credential creation and client use.
        
        # Create credentials object using potentially stale token
        credentials = {
            'aws_access_key_id': AWS_ACCESS_KEY_ID,
            'aws_secret_access_key': AWS_SECRET_ACCESS_KEY,
            'aws_session_token': AWS_SESSION_TOKEN  # Token from config block above
        }
        
        # Override token (simulating a 'security cleanup' step that happens later)
        AWS_SESSION_TOKEN = ""  # This line is the bug: it reassigns the global, but credentials dict already captured the old token.
        
        # Create STS client with the captured credentials (which have an invalidated token)
        sts_client = boto3.client(
            'sts',
            aws_access_key_id=credentials['aws_access_key_id'],
            aws_secret_access_key=credentials['aws_secret_access_key'],
            aws_session_token=credentials['aws_session_token']  # Uses old token, now mismatched
        )
        
        response = sts_client.get_caller_identity()
        print(f"Caller ARN: {response['Arn']}")
        return True
    except Exception as e:
        print(f"ERROR: Failed to verify caller identity: {e}", file=sys.stderr)
        # TODO: Add retry logic with exponential backoff (JIRA: OPS-2211)
        return False

if __name__ == "__main__":
    # This is called by the legacy cron job on the 'my-registry.internal/legacy-api' container
    success = verify_caller_identity()
    sys.exit(0 if success else 1)
AI Audit Console
> Waiting for pre-flight audit...
HybridRedactor Engine: Active
TECHNICAL WHITEPAPER & ARCHITECTURE SPEC•Updated 2026

Fixing AWS STS GetCallerIdentity 'Invalid Security Token' Error for Temporary Credentials

Threat/Impact Level: HIGH | Exploitability/Downtime Risk: HIGH | Time to Fix: 5–15 mins

TL;DR

  • What broke: sts:GetCallerIdentity rejected your temporary credentials because the X-Amz-Security-Token is expired, malformed, revoked, or the system clock is skewed beyond AWS's 5-minute tolerance.
  • How to fix it: Reissue the temporary credentials via sts:AssumeRole or refresh the instance/ECS/EKS metadata endpoint; verify NTP sync; confirm the session token is passed as the third credential parameter, not omitted.
  • Shortcut: Use our Client-Side Sandbox above to auto-refactor your credential initialization code — no data leaves your browser.

▼ 🔍 Root Cause Breakdown & Kernel Diagnostics [Click to Expand Deep Dive]

The Incident (What Does the Error Mean?)

Raw error output from AWS CLI or SDK:

An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation:
The request was rejected because the security token is invalid.

or via Boto3/SDK:

botocore.exceptions.ClientError: An error occurred (InvalidClientTokenId):
The security token included in the request is invalid.

Immediate consequence: Every downstream call gated on identity verification fails. This includes service-to-service auth handshakes, Vault AWS auth backends, IRSA token validation in EKS, and any CI/CD pipeline step that assumes a role before deploying. The pipeline is dead until credentials are valid.


▼ 🔍 Detailed Impact & Blast Radius Analysis [Click to Expand Deep Dive]

The Attack Vector / Blast Radius

This error is not just an annoyance — it signals a credential lifecycle failure that has real security implications:

  • If the token was revoked: Someone with IAM write access may have called sts:RevokeSession or deleted the underlying IAM role trust policy. Treat this as a potential credential compromise event until confirmed otherwise. Check CloudTrail immediately for DeleteRole, UpdateAssumeRolePolicy, or RevokeSession events.
  • If the token is expired and your code silently retries with stale credentials: Depending on your SDK retry logic, you may be logging the full AWS_SESSION_TOKEN value in retry error traces — exposing a still-valid-until-recently token in plaintext logs.
  • Clock skew >5 minutes: AWS SigV4 signing rejects requests outside a ±5-minute window. On EC2 instances with chrony misconfigured, or in air-gapped environments, this silently breaks all IAM auth. An attacker who can manipulate the system clock on a compromised host can deliberately push it outside the window to deny service.
  • Blast radius: Any service using this identity — S3 bucket access, Secrets Manager reads, ECR pulls, DynamoDB operations — is fully blocked. In a microservices mesh where one service assumes a role and vends sub-tokens, the failure cascades to every dependent service.

How to Fix It (The Solution)

Root Cause Checklist (run in this order)

  1. Check token expiry — STS temporary credentials expire between 15 minutes and 36 hours depending on DurationSeconds at assume-role time.
  2. Verify clock sync — timedatectl status or chronyc tracking. Offset must be <5 minutes.
  3. Confirm session token is present — Temporary credentials require three components: AccessKeyId, SecretAccessKey, AND SessionToken. Omitting the session token is the #1 cause of this error.
  4. Check partition/region — A token issued in aws-cn or aws-us-gov is invalid in the standard aws partition.
  5. Check CloudTrail for revocation — RevokeSession, DeleteRole, DetachRolePolicy events in the last hour.

Basic Fix — Boto3 (Python)

import boto3

- # WRONG: Omitting session token for temporary credentials
- client = boto3.client(
-     'sts',
-     aws_access_key_id='ASIA....',
-     aws_secret_access_key='wJalrXUtn....',
- )

+ # CORRECT: All three components required for temporary credentials
+ client = boto3.client(
+     'sts',
+     aws_access_key_id='ASIA....',
+     aws_secret_access_key='wJalrXUtn....',
+     aws_session_token='IQoJb3JpZ2lu...',  # Required for STS-issued creds
+ )

response = client.get_caller_identity()

Basic Fix — AWS CLI credential refresh

- # Stale assume-role output hardcoded in ~/.aws/credentials
- [default]
- aws_access_key_id = ASIA...OLD
- aws_secret_access_key = oldSecret
- # aws_session_token line missing entirely

+ # Re-issue via CLI and export fresh credentials
+ export $(aws sts assume-role \
+   --role-arn arn:aws:iam::123456789012:role/MyRole \
+   --role-session-name debug-session \
+   --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' \
+   --output text | awk '{print "AWS_ACCESS_KEY_ID="$1" AWS_SECRET_ACCESS_KEY="$2" AWS_SESSION_TOKEN="$3}')

Enterprise Best Practice — Auto-Refresh with SDK Credential Providers

Never hardcode or manually manage temporary credentials in production. Use the SDK's built-in provider chain, which handles refresh automatically.

- # ANTI-PATTERN: Manually managing short-lived credentials in application code
- creds = get_creds_from_vault()  # Fetched once at startup, never refreshed
- client = boto3.client('sts',
-     aws_access_key_id=creds['AccessKeyId'],
-     aws_secret_access_key=creds['SecretAccessKey'],
-     aws_session_token=creds['SessionToken']
- )

+ # BEST PRACTICE: Use AssumeRoleProvider via botocore RefreshableCredentials
+ from botocore.credentials import RefreshableCredentials
+ from botocore.session import get_session
+ from datetime import datetime, timezone
+
+ def refresh_credentials():
+     sts = boto3.client('sts')
+     response = sts.assume_role(
+         RoleArn='arn:aws:iam::123456789012:role/MyRole',
+         RoleSessionName='auto-refresh-session',
+         DurationSeconds=3600
+     )
+     c = response['Credentials']
+     return {
+         'access_key': c['AccessKeyId'],
+         'secret_key': c['SecretAccessKey'],
+         'token': c['SessionToken'],
+         'expiry_time': c['Expiration'].isoformat()
+     }
+
+ refreshable = RefreshableCredentials.create_from_metadata(
+     metadata=refresh_credentials(),
+     refresh_using=refresh_credentials,
+     method='sts-assume-role'
+ )

For EKS/IRSA: Ensure AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN env vars are injected by the pod identity webhook. The SDK handles the rest.

For EC2/ECS: Do not set AWS_ACCESS_KEY_ID manually. Delete the env var and let the IMDSv2 metadata provider serve auto-rotating credentials.


💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.

(For CI/CD pipeline automation, install via npm i -g stackengine).


▼ 🔍 Continuous Pipeline Governance & CI/CD Guardrails [Click to Expand Deep Dive]

Prevention in CI/CD

1. Enforce credential expiry checks with Checkov

# .checkov.yml — flag any hardcoded AWS credential patterns
checks:
  - CKV_AWS_46  # Ensure no hardcoded credentials in Lambda env vars
  - CKV_SECRET_2 # AWS Secret Access Key in source

2. OPA policy — block static credentials in ECS task definitions

package aws.ecs

deny[msg] {
  container := input.containerDefinitions[_]
  env := container.environment[_]
  regex.match(`^AWS_(ACCESS_KEY|SECRET|SESSION_TOKEN)`, env.name)
  msg := sprintf("Container '%v' hardcodes AWS credentials. Use task IAM role instead.", [container.name])
}

3. GitHub Actions — use OIDC, never store STS tokens as secrets

- # WRONG: Storing long-lived or manually-refreshed STS tokens in GitHub Secrets
- env:
-   AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
-   AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
-   AWS_SESSION_TOKEN: ${{ secrets.AWS_SESSION_TOKEN }}  # Expires, breaks pipeline

+ # CORRECT: GitHub OIDC federation — zero stored secrets, auto-rotating
+ permissions:
+   id-token: write
+   contents: read
+ steps:
+   - uses: aws-actions/configure-aws-credentials@v4
+     with:
+       role-to-assume: arn:aws:iam::123456789012:role/GitHubActionsRole
+       aws-region: us-east-1

4. NTP enforcement via AWS Config rule

Enable the ec2-instance-profile-attached and monitor for clock drift using CloudWatch agent metrics (chrony_tracking_system_time_offset). Alert if offset exceeds 240 seconds — before AWS's 300-second hard cutoff.

# Terraform — CloudWatch alarm for NTP drift
resource "aws_cloudwatch_metric_alarm" "ntp_drift" {
  alarm_name          = "ntp-offset-critical"
  metric_name         = "chrony_tracking_system_time_offset"
  namespace           = "CWAgent"
  statistic           = "Maximum"
  period              = 60
  evaluation_periods  = 2
  threshold           = 240
  comparison_operator = "GreaterThanThreshold"
  alarm_actions       = [aws_sns_topic.ops_alerts.arn]
}

Official Grounding & Normative Specifications

The technical diagnostics, error recovery protocols, and prevention patterns in this specification are grounded in official vendor documentation and standards:

Diagnostic & Architectural FAQ

Q1.Why does GetCallerIdentity return 'invalid security token' even when my credentials look correct?
The root cause is a mismatch in the AWS SigV4 signing components. Temporary credentials (ASIA-prefixed) are a triad: AccessKeyId, SecretAccessKey, and SessionToken. Omitting the SessionToken—often due to unexported environment variables or misconfigured credential chains—breaks the signature. Furthermore, AWS enforces a strict 5-minute clock skew tolerance; a desynchronized system clock will cause immediate rejection. This is a foundational identity verification failure that exposes your pipeline to authentication blind spots.
Q2.How do I tell if my STS token was revoked versus simply expired?
Manually distinguishing revocation from expiration is a high-risk, error-prone forensic exercise. You must cross-reference CloudTrail for 'RevokeSession', 'DeleteRole', or 'UpdateAssumeRolePolicy' events, while simultaneously checking IAM role tags and trust policies—a process prone to oversight and delay. Each minute of misdiagnosis expands the security blast radius, leaving roles or resources exposed. Instead of manual investigation, drop the broken manifest into the StackEngine Local Sandbox above to automatically compile a zero-trust fix that isolates the compromised identity and generates a secure, least-privilege replacement.
Q3.Can this error occur on EC2 or ECS instances that use instance profiles?
Absolutely, and at scale, it becomes a compliance nightmare. On EC2 or ECS, environment variables (AWS_ACCESS_KEY_ID, AWS_SESSION_TOKEN) override the secure, auto-rotating IMDSv2 endpoint. Manual credential leakage or stale env vars persist across deployments, creating shadow identities that evade rotation policies. Managing this across large mono-repos or hundreds of instances is operationally unsustainable and fails audit controls. For automated pull request generation with Maker-Checker review, upgrade to the StackEngine CI/CD Automator (Team Edition), which enforces identity hygiene, eliminates credential drift, and generates compliance-proof manifests across your entire infrastructure.