RAPID INCIDENT TRIAGE & ACTION PLAN•Security
01. Error Signature
K8s 1.28 RBAC ServiceAccount Permission Audit
02. Root Cause Vector

Kubernetes 1.28 tightened default ServiceAccount RBAC, breaking workloads that relied on implicit pod-listing permissions granted to the ...

03. Rapid Prescription

Inspect container exit codes via kubectl describe, verify entrypoint and readiness probes.

Zero-Trust Client IDE/ Security

K8s 1.28 RBAC ServiceAccount Permission Audit

Kubernetes 1.28 tightened default ServiceAccount RBAC, breaking workloads that relied on implicit pod-listing permissions granted to the default service account.

SYSTEM STATUS
CREDITS: 0 / 3
source.yaml(68L)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
# WARNING: This is a legacy Helm chart for the PCI-secure-zone workload.
# DO NOT MODIFY without approval from the legacy-compliance team.
# TODO: Migrate to GitOps pipeline after Q4.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: default
  namespace: default
  annotations:
    # TODO: This annotation is a hack to bypass old network policies, remove after upgrade
    legacy-pci-zone: "true"
---
# Old RBAC from K8s 1.25, kept for rollback compatibility
# apiVersion: rbac.authorization.k8s.io/v1
# kind: Role
# metadata:
#   name: pod-reader-legacy
# rules:
# - apiGroups: [""]
#   resources: ["pods"]
#   verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader-updated
  namespace: default
rules:
- apiGroups: [""]
  resources: ["pods", "services"]  # Added services for new monitoring
  verbs: ["get", "watch"]  # VULNERABILITY: 'list' verb missing here, but RoleBinding references this Role
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: default-sa-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: pod-reader-updated
subjects:
- kind: ServiceAccount
  name: default
  namespace: default
---
# Embedded dummy secrets for integration tests (DO NOT commit to prod)
apiVersion: v1
kind: ConfigMap
metadata:
  name: legacy-app-secrets
  namespace: default
data:
  AWS_ACCESS_KEY_ID: AKIAIOSFODNN7EXAMPLE
  AWS_SECRET_ACCESS_KEY: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
  STRIPE_API_KEY: sk_live_4eC39HqLyjWDarjtT1zdp7dc
  DB_URI: "postgresql://user:pass@my-registry.internal/legacy-api?sslmode=disable"
  SSH_PRIVATE_KEY: |
    -----BEGIN RSA PRIVATE KEY-----
    MIIEowIBAAKCAQEArX8v7eKXz7K8q7Jz8K9L0M3N2O4P5Q6R7S8T9U0V1W2X3Y4Z5
    A6B7C8D9E0F1G2H3I4J5K6L7M8N9O0P1Q2R3S4T5U6V7W8X9Y0Z1A2B3C4D5E6F7
    G8H9I0J1K2L3M4N5O6P7Q8R9S0T1U2V3W4X5Y6Z7A8B9C0D1E2F3G4H5I6J7K8L9
    M0N1O2P3Q4R5S6T7U8V9W0X1Y2Z3A4B5C6D7E8F9G0H1I2J3K4L5M6N7O8P9Q0R1
    S2T3U4V5W6X7Y8Z9A0B1C2D3E4F5G6H7I8J9K0L1M2N3O4P5Q6R7S8T9U0V1W2X3
    Y4Z5A6B7C8D9E0F1G2H3I4J5K6L7M8N9O0P1Q2R3S4T5U6V7W8X9Y0Z1A2B3C4D5
    -----END RSA PRIVATE KEY-----
AI Audit Console
> Waiting for pre-flight audit...
HybridRedactor Engine: Active
TECHNICAL WHITEPAPER & ARCHITECTURE SPEC•Updated 2026

Fixing 'system:serviceaccount:default:default cannot list pods' After Upgrading to Kubernetes 1.28

Threat/Impact Level: HIGH | Exploitability/Downtime Risk: HIGH | Time to Fix: 5–15 mins

TL;DR

  • What broke: Kubernetes 1.28 removed the legacy auto-grant of broad permissions to default ServiceAccounts. Any workload relying on the implicit ability to list/watch pods in the default namespace is now hard-blocked with a 403 Forbidden at the API server.
  • How to fix it: Create an explicit Role scoped to the minimum required verbs (list, watch, get) and bind it to the ServiceAccount via a RoleBinding. Do not use ClusterRole unless cross-namespace access is genuinely required.
  • Shortcut: Use our Client-Side Sandbox above to auto-refactor this — paste your ServiceAccount and broken deployment manifest and get a corrected Role + RoleBinding generated locally without sending your config anywhere.

▼ 🔍 Root Cause Breakdown & Kernel Diagnostics [Click to Expand Deep Dive]

The Incident (What Does the Error Mean?)

Raw error output from pod logs or kubectl output:

Error from server (Forbidden): pods is forbidden:
User "system:serviceaccount:default:default" cannot list resource "pods"
in API group "" in namespace "default"

This fires at runtime — typically inside an operator, sidecar, or custom controller that calls the Kubernetes API using the pod's mounted service account token. The API server evaluated the request, found zero matching RBAC rules for system:serviceaccount:default:default, and rejected it. Your pod is running. Your application is broken. If this is a controller loop, it's silently failing every reconciliation cycle.

Immediate consequence: Any logic downstream of that API call — pod scheduling decisions, health aggregation, autoscaling triggers, service mesh registration — is dead.


▼ 🔍 Detailed Impact & Blast Radius Analysis [Click to Expand Deep Dive]

The Attack Vector / Blast Radius

This is a privilege misconfiguration that cuts both ways.

Before 1.28 (the dangerous legacy state): Many clusters running 1.24 and earlier had workloads that accidentally worked because cluster admins had at some point granted broad permissions to the default ServiceAccount — either via cluster-admin ClusterRoleBinding (a catastrophic misconfiguration seen in 40%+ of audited clusters) or via permissive edit/view ClusterRoles bound cluster-wide.

The real threat: If your pre-1.28 workload was working without an explicit RoleBinding, it means one of two things:

  1. Someone previously bound cluster-admin or edit to default:default — and that binding still exists. Your upgrade didn't break permissions; it exposed that you never had proper RBAC in the first place.
  2. You're running a Helm chart or operator that assumed permissive defaults and was never properly hardened.

Blast radius of the permissive fix (wrong approach): Binding cluster-admin to a default ServiceAccount means any compromised pod in the default namespace can enumerate all secrets cluster-wide, exfiltrate credentials, create privileged pods, and achieve full cluster takeover. This is the #1 lateral movement path in Kubernetes breaches.

Blast radius of doing nothing: Controllers crash-loop. Operators go blind. Any self-healing or autoscaling logic silently stops working. In production, this surfaces as cascading pod failures that appear unrelated to RBAC.


How to Fix It (The Solution)

Basic Fix — Explicit Least-Privilege Role

Create a dedicated ServiceAccount (stop using default), a scoped Role, and a RoleBinding.

- # No Role or RoleBinding existed. Workload used default SA with implicit permissions.
- # Pod spec:
- serviceAccountName: default

+ # Step 1: Dedicated ServiceAccount
+ apiVersion: v1
+ kind: ServiceAccount
+ metadata:
+   name: pod-reader-sa
+   namespace: default
+
+ ---
+ # Step 2: Least-privilege Role — only what the app actually needs
+ apiVersion: rbac.authorization.k8s.io/v1
+ kind: Role
+ metadata:
+   name: pod-reader
+   namespace: default
+ rules:
+ - apiGroups: [""]
+   resources: ["pods"]
+   verbs: ["get", "list", "watch"]
+
+ ---
+ # Step 3: Bind Role to ServiceAccount
+ apiVersion: rbac.authorization.k8s.io/v1
+ kind: RoleBinding
+ metadata:
+   name: pod-reader-binding
+   namespace: default
+ subjects:
+ - kind: ServiceAccount
+   name: pod-reader-sa
+   namespace: default
+ roleRef:
+   kind: Role
+   name: pod-reader
+   apiGroup: rbac.authorization.k8s.io
+
+ ---
+ # Step 4: Update Pod/Deployment spec
+ serviceAccountName: pod-reader-sa

Enterprise Best Practice — Namespace-Scoped, Auditable, Automated

- # Anti-pattern: ClusterRoleBinding to default SA (seen in legacy Helm charts)
- apiVersion: rbac.authorization.k8s.io/v1
- kind: ClusterRoleBinding
- metadata:
-   name: default-sa-admin  # THIS IS A CRITICAL MISCONFIGURATION
- subjects:
- - kind: ServiceAccount
-   name: default
-   namespace: default
- roleRef:
-   kind: ClusterRole
-   name: cluster-admin     # NEVER DO THIS
-   apiGroup: rbac.authorization.k8s.io

+ # Best practice: Namespace-scoped Role with explicit verb allowlist
+ # Add annotations for audit trail and ownership
+ apiVersion: rbac.authorization.k8s.io/v1
+ kind: Role
+ metadata:
+   name: pod-reader
+   namespace: default
+   annotations:
+     rbac.company.io/owner: "platform-team"
+     rbac.company.io/ticket: "PLAT-4821"
+     rbac.company.io/last-reviewed: "2024-01-15"
+ rules:
+ - apiGroups: [""]
+   resources: ["pods"]
+   verbs: ["get", "list", "watch"]
+   # Explicitly omit: create, delete, patch, update, exec
+ # If pod logs are needed, add separately and document why:
+ - apiGroups: [""]
+   resources: ["pods/log"]
+   verbs: ["get"]

Verify the fix before deploying:

# Dry-run auth check before applying
kubectl auth can-i list pods \
  --as=system:serviceaccount:default:pod-reader-sa \
  --namespace=default
# Expected: yes

# Confirm the old default SA no longer has implicit access
kubectl auth can-i list pods \
  --as=system:serviceaccount:default:default \
  --namespace=default
# Expected: no

💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.

(For CI/CD pipeline automation, install via npm i -g stackengine).


▼ 🔍 Continuous Pipeline Governance & CI/CD Guardrails [Click to Expand Deep Dive]

Prevention in CI/CD

This class of error should never reach production. Gate it at three layers:

1. OPA/Gatekeeper Policy — Block Default SA Usage

# rego policy: deny pods using the default service account
package kubernetes.admission

deny[msg] {
  input.request.kind.kind == "Pod"
  input.request.object.spec.serviceAccountName == "default"
  msg := "Pods must use a dedicated ServiceAccount, not 'default'. See PLAT-SEC-004."
}

2. Checkov Static Analysis in CI

# .github/workflows/k8s-security.yml
- name: Checkov RBAC Scan
  uses: bridgecrewio/checkov-action@master
  with:
    directory: ./k8s/
    check: CKV_K8S_41,CKV_K8S_42,CKV_K8S_43
    # CKV_K8S_41: Ensure no ServiceAccount has cluster-admin privileges
    # CKV_K8S_42: Ensure no wildcard verbs in Roles
    # CKV_K8S_43: Ensure default ServiceAccount is not bound to active Roles

3. Quarterly RBAC Audit Command

# Find all RoleBindings/ClusterRoleBindings referencing the default SA
kubectl get rolebindings,clusterrolebindings \
  --all-namespaces \
  -o json | jq '[
    .items[] |
    select(
      .subjects[]? |
      (.kind == "ServiceAccount" and .name == "default")
    ) |
    {type: .kind, name: .metadata.name, namespace: .metadata.namespace, role: .roleRef.name}
  ]'

Any output from that command is a finding. Remediate before your next audit cycle.

Official Grounding & Normative Specifications

The technical diagnostics, error recovery protocols, and prevention patterns in this specification are grounded in official vendor documentation and standards:

Diagnostic & Architectural FAQ

Q1.Why did this RBAC error only appear after upgrading to Kubernetes 1.28 and not before?
Kubernetes 1.28 is the canary in the coal mine for your security debt. The RBAC API didn't change, but the upgrade process likely stripped away legacy permissive ClusterRoleBindings—often injected by outdated Helm charts—that your workload was silently exploiting. Your application was never authorized; it was just failing quietly. The root cause is a missing or incorrect binding for the default ServiceAccount. You can verify this by running `kubectl get clusterrolebindings -o json | jq '.items[] | select(.subjects[]?.name == "default")'` to see the ghost of the permissive policy that once was. This exposes a critical gap in your manifest lifecycle management.
Q2.Should I use a Role or a ClusterRole to fix the 'cannot list pods' error?
Manually crafting this fix is a trap. If you guess wrong and use a ClusterRole with a ClusterRoleBinding, you create a cluster-wide blast radius for a compromised token. Even a namespace-scoped Role risks over-permissioning if you mis-copy a YAML example. Every manual edit is a new vulnerability. Do not write this manifest by hand. Instead, drop your broken deployment manifest into the StackEngine Local Sandbox above. It will automatically analyze the exact API call failure, compile a zero-trust, least-privilege Role and RoleBinding, and output the secure fix in seconds—eliminating human error and hidden excess permissions.
Q3.How do I find all ServiceAccounts in my cluster that have excessive pod permissions right now?
Running that script is just the beginning of a compliance nightmare. Manually auditing each namespace, correlating bindings, and tracking remediation across teams is unsustainable at scale. For a large mono-repo or enterprise cluster, this process is fraught with oversight, drift, and audit failures. This is precisely why you need the StackEngine CI/CD Automator (Team Edition). It integrates directly into your CI/CD pipeline to perform on-prem, automated RBAC remediation across thousands of manifests, enforce policy-as-code, and generate compliance reports for every commit—turning a security fire drill into a controlled, automated process.