Fixing EKS IRSA AccessDenied: Missing or Invalid OIDC Provider Thumbprint
Threat/Impact Level: HIGH | Downtime Risk: HIGH | Time to Fix: 5–15 mins
TL;DR
- What broke: Your EKS pod's service account cannot assume its IAM role because AWS STS rejected the OIDC token — the OIDC provider in IAM is either missing its TLS thumbprint, has a stale one after an AWS-side certificate rotation, or the issuer URL has a trailing slash mismatch.
- How to fix it: Re-derive the OIDC provider thumbprint from the live endpoint, update the IAM OIDC provider resource, and verify the trust policy
StringEqualscondition matches the exact service account ARN format. - Fast path: Use our Client-Side Sandbox above to auto-refactor your trust policy and OIDC provider Terraform block without leaking your ARNs.
▼ 🔍 Root Cause Breakdown & Kernel Diagnostics [Click to Expand Deep Dive]
The Incident (What does the error mean?)
Your pod logs or kubectl describe pod surfaces this:
An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation:
Not authorized to perform sts:AssumeRoleWithWebIdentityOr from the AWS SDK inside the container:
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the
AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentityImmediate consequence: Every pod using this service account loses all AWS API access. If this is your logging, secrets, or storage workload — it is down. STS is not retrying. The token is being presented, but the OIDC provider record in IAM cannot be validated because the thumbprint chain is broken or absent.
▼ 🔍 Detailed Impact & Blast Radius Analysis [Click to Expand Deep Dive]
The Attack Vector / Blast Radius
This is not just an ops inconvenience. The OIDC thumbprint is the cryptographic anchor that tells AWS IAM: "trust JWTs signed by this OIDC issuer." When it's wrong or missing:
- All pods bound to any service account using this OIDC provider lose IAM access simultaneously — not just one pod, the entire provider is untrusted.
- A stale thumbprint (AWS rotated the EKS OIDC endpoint certificate — this has happened in production) causes a silent, total IRSA failure across your cluster with no pre-warning.
- If you "fix" this by broadening the trust policy to remove the
StringEqualscondition onsub(the service account), you've created a privilege escalation vector: any pod in the cluster that can obtain an OIDC token can now assume the role. - Misconfigured thumbprints are frequently the result of copy-pasting an OIDC provider ARN from a different cluster/region — the issuer URL looks identical but the thumbprint is environment-specific.
How to Fix It (The Solution)
Step 1: Re-derive the live thumbprint
Do not guess or copy from docs. Derive it from the live endpoint:
# Get your cluster's OIDC issuer URL
aws eks describe-cluster --name <cluster-name> --query "cluster.identity.oidc.issuer" --output text
# Example output: https://oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE
# Extract the thumbprint (last cert in chain, SHA-1 fingerprint)
OIDC_URL=$(aws eks describe-cluster --name <cluster-name> \
--query "cluster.identity.oidc.issuer" --output text | sed 's|https://||')
openssl s_client -servername $OIDC_URL -showcerts \
-connect ${OIDC_URL}:443 </dev/null 2>/dev/null \
| awk '/-----BEGIN CERTIFICATE-----/{c++} c==2{print}' \
| openssl x509 -fingerprint -noout -sha1 \
| sed 's/://g' | awk -F= '{print tolower($2)}'⚠️ You must target the second-to-last (root CA) certificate in the chain, not the leaf. The
c==2awk filter handles this. AWS docs confirm SHA-1 of the root CA is what IAM validates.
Basic Fix: Update the OIDC Provider thumbprint via CLI
aws iam update-open-id-connect-provider-thumbprint \
--open-id-connect-provider-arn arn:aws:iam::<account-id>:oidc-provider/oidc.eks.<region>.amazonaws.com/id/<oidc-id> \
--thumbprint-list <new-thumbprint-hex>Enterprise Best Practice: Terraform with locked thumbprint + least-privilege trust policy
- resource "aws_iam_openid_connect_provider" "eks" {
- url = data.aws_eks_cluster.main.identity[0].oidc[0].issuer
- client_id_list = ["sts.amazonaws.com"]
- # thumbprint_list omitted — THIS IS THE BUG
- }
+ resource "aws_iam_openid_connect_provider" "eks" {
+ url = data.aws_eks_cluster.main.identity[0].oidc[0].issuer
+ client_id_list = ["sts.amazonaws.com"]
+ thumbprint_list = [data.tls_certificate.eks_oidc.certificates[0].sha1_fingerprint]
+ }
+
+ data "tls_certificate" "eks_oidc" {
+ url = data.aws_eks_cluster.main.identity[0].oidc[0].issuer
+ }And the trust policy — never omit the sub condition:
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::<account>:oidc-provider/oidc.eks.<region>.amazonaws.com/id/<id>"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
- "oidc.eks.<region>.amazonaws.com/id/<id>:aud": "sts.amazonaws.com"
- // Missing sub condition — ANY pod can assume this role
+ "oidc.eks.<region>.amazonaws.com/id/<id>:aud": "sts.amazonaws.com",
+ "oidc.eks.<region>.amazonaws.com/id/<id>:sub": "system:serviceaccount:<namespace>:<serviceaccount-name>"
}
}
}💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.
(For CI/CD pipeline automation, install via
npm i -g stackengine).
▼ 🔍 Continuous Pipeline Governance & CI/CD Guardrails [Click to Expand Deep Dive]
Prevention in CI/CD
1. Checkov — catch missing thumbprint at plan time
Checkov rule CKV_AWS_79 flags OIDC providers with empty thumbprint_list. Add to your pipeline:
checkov -d . --check CKV_AWS_79 --framework terraform2. OPA/Conftest policy — enforce `sub` condition in trust policies
# policy/irsa_trust.rego
package irsa
deny[msg] {
doc := input.resource_changes[_]
doc.type == "aws_iam_role"
assume := doc.change.after.assume_role_policy
policy := json.unmarshal(assume)
stmt := policy.Statement[_]
stmt.Action == "sts:AssumeRoleWithWebIdentity"
not _has_sub_condition(stmt)
msg := sprintf("Role %v is missing sub condition on IRSA trust policy", [doc.address])
}
_has_sub_condition(stmt) {
keys := object.keys(stmt.Condition.StringEquals)
endswith(keys[_], ":sub")
}3. Renovate / scheduled Terraform drift detection
The EKS OIDC endpoint certificate can rotate without notice. Run terraform plan on a schedule (daily) in your CI. A thumbprint drift will surface as a planned change before it causes an outage:
# .github/workflows/drift-detect.yml
schedule:
- cron: '0 6 * * *'
jobs:
drift:
steps:
- run: terraform plan -detailed-exitcodeExitcode 2 = drift detected. Alert on it.
Official Grounding & Normative Specifications
The technical diagnostics, error recovery protocols, and prevention patterns in this specification are grounded in official vendor documentation and standards:
Federal security guidelines for secret exfiltration prevention, automated integrity checks, and peer sign-offs.
Core memory subsystem controls, pressure stall information (PSI), and kernel OOM invocation mechanics.
Industry-standard specification for container execution boundaries and POSIX namespace isolation.