Fixing 'system:serviceaccount:default:default cannot list pods' After Upgrading to Kubernetes 1.28

Threat/Impact Level: HIGH | Exploitability/Downtime Risk: HIGH | Time to Fix: 5–15 mins

TL;DR

• What broke: Kubernetes 1.28 removed the legacy auto-grant of broad permissions to default ServiceAccounts. Any workload relying on the implicit ability to list/watch pods in the default namespace is now hard-blocked with a 403 Forbidden at the API server.
• How to fix it: Create an explicit Role scoped to the minimum required verbs (list, watch, get) and bind it to the ServiceAccount via a RoleBinding. Do not use ClusterRole unless cross-namespace access is genuinely required.
• Shortcut: Use our Client-Side Sandbox below to auto-refactor this — paste your ServiceAccount and broken deployment manifest and get a corrected Role + RoleBinding generated locally without sending your config anywhere.

The Incident (What Does the Error Mean?)

Raw error output from pod logs or kubectl output:

Error from server (Forbidden): pods is forbidden:
User "system:serviceaccount:default:default" cannot list resource "pods"
in API group "" in namespace "default"

This fires at runtime — typically inside an operator, sidecar, or custom controller that calls the Kubernetes API using the pod's mounted service account token. The API server evaluated the request, found zero matching RBAC rules for system:serviceaccount:default:default, and rejected it. Your pod is running. Your application is broken. If this is a controller loop, it's silently failing every reconciliation cycle.

Immediate consequence: Any logic downstream of that API call — pod scheduling decisions, health aggregation, autoscaling triggers, service mesh registration — is dead.

The Attack Vector / Blast Radius

This is a privilege misconfiguration that cuts both ways.

Before 1.28 (the dangerous legacy state): Many clusters running 1.24 and earlier had workloads that accidentally worked because cluster admins had at some point granted broad permissions to the default ServiceAccount — either via cluster-admin ClusterRoleBinding (a catastrophic misconfiguration seen in 40%+ of audited clusters) or via permissive edit/view ClusterRoles bound cluster-wide.

The real threat: If your pre-1.28 workload was working without an explicit RoleBinding, it means one of two things:

1. Someone previously bound cluster-admin or edit to default:default — and that binding still exists. Your upgrade didn't break permissions; it exposed that you never had proper RBAC in the first place.
2. You're running a Helm chart or operator that assumed permissive defaults and was never properly hardened.

Blast radius of the permissive fix (wrong approach): Binding cluster-admin to a default ServiceAccount means any compromised pod in the default namespace can enumerate all secrets cluster-wide, exfiltrate credentials, create privileged pods, and achieve full cluster takeover. This is the #1 lateral movement path in Kubernetes breaches.

Blast radius of doing nothing: Controllers crash-loop. Operators go blind. Any self-healing or autoscaling logic silently stops working. In production, this surfaces as cascading pod failures that appear unrelated to RBAC.

How to Fix It (The Solution)

Basic Fix — Explicit Least-Privilege Role

Create a dedicated ServiceAccount (stop using default), a scoped Role, and a RoleBinding.

- # No Role or RoleBinding existed. Workload used default SA with implicit permissions.
- # Pod spec:
- serviceAccountName: default

+ # Step 1: Dedicated ServiceAccount
+ apiVersion: v1
+ kind: ServiceAccount
+ metadata:
+   name: pod-reader-sa
+   namespace: default
+
+ ---
+ # Step 2: Least-privilege Role — only what the app actually needs
+ apiVersion: rbac.authorization.k8s.io/v1
+ kind: Role
+ metadata:
+   name: pod-reader
+   namespace: default
+ rules:
+ - apiGroups: [""]
+   resources: ["pods"]
+   verbs: ["get", "list", "watch"]
+
+ ---
+ # Step 3: Bind Role to ServiceAccount
+ apiVersion: rbac.authorization.k8s.io/v1
+ kind: RoleBinding
+ metadata:
+   name: pod-reader-binding
+   namespace: default
+ subjects:
+ - kind: ServiceAccount
+   name: pod-reader-sa
+   namespace: default
+ roleRef:
+   kind: Role
+   name: pod-reader
+   apiGroup: rbac.authorization.k8s.io
+
+ ---
+ # Step 4: Update Pod/Deployment spec
+ serviceAccountName: pod-reader-sa

Enterprise Best Practice — Namespace-Scoped, Auditable, Automated

- # Anti-pattern: ClusterRoleBinding to default SA (seen in legacy Helm charts)
- apiVersion: rbac.authorization.k8s.io/v1
- kind: ClusterRoleBinding
- metadata:
-   name: default-sa-admin  # THIS IS A CRITICAL MISCONFIGURATION
- subjects:
- - kind: ServiceAccount
-   name: default
-   namespace: default
- roleRef:
-   kind: ClusterRole
-   name: cluster-admin     # NEVER DO THIS
-   apiGroup: rbac.authorization.k8s.io

+ # Best practice: Namespace-scoped Role with explicit verb allowlist
+ # Add annotations for audit trail and ownership
+ apiVersion: rbac.authorization.k8s.io/v1
+ kind: Role
+ metadata:
+   name: pod-reader
+   namespace: default
+   annotations:
+     rbac.company.io/owner: "platform-team"
+     rbac.company.io/ticket: "PLAT-4821"
+     rbac.company.io/last-reviewed: "2024-01-15"
+ rules:
+ - apiGroups: [""]
+   resources: ["pods"]
+   verbs: ["get", "list", "watch"]
+   # Explicitly omit: create, delete, patch, update, exec
+ # If pod logs are needed, add separately and document why:
+ - apiGroups: [""]
+   resources: ["pods/log"]
+   verbs: ["get"]

Verify the fix before deploying:

# Dry-run auth check before applying
kubectl auth can-i list pods \
  --as=system:serviceaccount:default:pod-reader-sa \
  --namespace=default
# Expected: yes

# Confirm the old default SA no longer has implicit access
kubectl auth can-i list pods \
  --as=system:serviceaccount:default:default \
  --namespace=default
# Expected: no

💡 Tired of pasting proprietary configs into ChatGPT? Generic AI tools log your company's ARNs, DB strings, and private keys. StackEngine is a zero-backend, pure Client-Side WASM utility. Drop your failing config into the sandbox above. We redact your secrets locally in the browser and auto-generate the refactored code using your own API key.

Prevention in CI/CD

This class of error should never reach production. Gate it at three layers:

1. OPA/Gatekeeper Policy — Block Default SA Usage

# rego policy: deny pods using the default service account
package kubernetes.admission

deny[msg] {
  input.request.kind.kind == "Pod"
  input.request.object.spec.serviceAccountName == "default"
  msg := "Pods must use a dedicated ServiceAccount, not 'default'. See PLAT-SEC-004."
}

2. Checkov Static Analysis in CI

# .github/workflows/k8s-security.yml
- name: Checkov RBAC Scan
  uses: bridgecrewio/checkov-action@master
  with:
    directory: ./k8s/
    check: CKV_K8S_41,CKV_K8S_42,CKV_K8S_43
    # CKV_K8S_41: Ensure no ServiceAccount has cluster-admin privileges
    # CKV_K8S_42: Ensure no wildcard verbs in Roles
    # CKV_K8S_43: Ensure default ServiceAccount is not bound to active Roles

3. Quarterly RBAC Audit Command

# Find all RoleBindings/ClusterRoleBindings referencing the default SA
kubectl get rolebindings,clusterrolebindings \
  --all-namespaces \
  -o json | jq '[
    .items[] |
    select(
      .subjects[]? |
      (.kind == "ServiceAccount" and .name == "default")
    ) |
    {type: .kind, name: .metadata.name, namespace: .metadata.namespace, role: .roleRef.name}
  ]'

Any output from that command is a finding. Remediate before your next audit cycle.