EKS IRSA IAM Log Access Verifier

Threat/Impact Level: HIGH | Downtime Risk: HIGH | Time to Fix: 15–30 mins

TL;DR

• What broke: kubectl logs -f initiates a WebSocket upgrade request through the EKS API server to the kubelet. A 403 Forbidden means the IAM principal or the Kubernetes RBAC subject is being denied at one of three checkpoints: the aws-auth ConfigMap mapping, the RBAC pods/log verb, or the node IAM instance profile lacking AmazonEKSWorkerNodePolicy.
• How to fix it: Verify the aws-auth ConfigMap maps your IAM role correctly, confirm your RBAC ClusterRole includes pods/log and pods/exec verbs, and ensure the EC2 node instance profile has the required managed policies attached.
• Shortcut: Use our Client-Side Sandbox above to auto-refactor your aws-auth ConfigMap and RBAC manifests without leaking your ARNs.

$ kubectl logs -f my-pod-7d9f8b-xkz2p
error: unable to upgrade connection: Forbidden (user=arn:aws:iam::123456789012:assumed-role/my-role/session, verb=create, resource=nodes, subresource=proxy)

How to Fix It

Step 1 — Confirm Who the API Server Thinks You Are

kubectl auth whoami
# or for older clusters:
kubectl get pods --v=9 2>&1 | grep "Response Status"

If the returned username is system:anonymous, your IAM role is not in aws-auth. If it returns your role ARN but still 403s, it's RBAC.

Fix A — `aws-auth` ConfigMap (Basic Fix)

# kubectl edit configmap aws-auth -n kube-system
 apiVersion: v1
 kind: ConfigMap
 metadata:
   name: aws-auth
   namespace: kube-system
 data:
   mapRoles: |
-    - rolearn: arn:aws:iam::123456789012:role/my-dev-role
-      username: my-dev-user
-      groups:
-        - system:masters
+    - rolearn: arn:aws:iam::123456789012:role/my-dev-role
+      username: my-dev-user
+      groups:
+        - eks-log-viewers

Do not map developer roles to system:masters. That grants cluster-admin. Map to a scoped RBAC group.

Fix B — RBAC ClusterRole with `pods/log` (Enterprise Best Practice)

-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRole
-metadata:
-  name: eks-log-viewers
-rules:
-  - apiGroups: [""]
-    resources: ["pods"]
-    verbs: ["get", "list", "watch"]
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: eks-log-viewers
+rules:
+  - apiGroups: [""]
+    resources: ["pods"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: [""]
+    resources: ["pods/log"]
+    verbs: ["get", "list"]
+  - apiGroups: [""]
+    resources: ["nodes/proxy"]
+    verbs: ["get"]
---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: eks-log-viewers-binding
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: eks-log-viewers
+subjects:
+  - kind: Group
+    name: eks-log-viewers
+    apiGroup: rbac.authorization.k8s.io

Fix C — Node IAM Instance Profile (Managed Policies)

 # Terraform: aws_iam_role_policy_attachment for node group role
-resource "aws_iam_role_policy_attachment" "node_cni_only" {
-  role       = aws_iam_role.eks_node.name
-  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
-}
+resource "aws_iam_role_policy_attachment" "node_worker" {
+  role       = aws_iam_role.eks_node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_cni" {
+  role       = aws_iam_role.eks_node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
+}
+
+resource "aws_iam_role_policy_attachment" "node_ecr_readonly" {
+  role       = aws_iam_role.eks_node.name
+  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
+}

Fix D — IRSA OIDC Trust Policy Condition Mismatch

 {
   "Effect": "Allow",
   "Principal": {
     "Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
   },
   "Action": "sts:AssumeRoleWithWebIdentity",
   "Condition": {
     "StringEquals": {
-      "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:default:my-sa"
+      "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:my-namespace:my-sa",
+      "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com"
     }
   }
 }

Namespace mismatch in the OIDC sub claim is the #1 silent IRSA failure. The token is issued but STS rejects AssumeRoleWithWebIdentity, causing the pod to fall back to the node instance profile — or fail entirely.

💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.

(For CI/CD pipeline automation, install via npm i -g stackengine).

checkov -d ./terraform --check CKV_AWS_58,CKV_AWS_37
# CKV_AWS_58: EKS node group must have AmazonEKSWorkerNodePolicy

package k8s.awsauth

violation[{"msg": msg}] {
  binding := input.review.object.data.mapRoles
  contains(binding, "system:masters")
  msg := "Mapping IAM roles to system:masters is prohibited. Use scoped RBAC groups."
}

# Add to post-deploy validation step
kubectl auth can-i get pods/log \
  --namespace=production \
  --as=system:serviceaccount:production:my-sa
# Must return: yes

 resource "aws_launch_template" "eks_node" {
+  metadata_options {
+    http_endpoint               = "enabled"
+    http_tokens                 = "required"  # IMDSv2 only
+    http_put_response_hop_limit = 1
+  }
 }

# Alert on any ConfigMap update in kube-system
aws events put-rule --name detect-aws-auth-mutation \
  --event-pattern '{"source":["aws.eks"],"detail-type":["AWS API Call via CloudTrail"],"detail":{"eventName":["UpdateConfigMap"],"requestParameters":{"namespace":["kube-system"],"name":["aws-auth"]}}}'