Why did my pods work on Kubernetes 1.23 but break after upgrading to 1.24+?

Kubernetes 1.24 implemented KEP-2799, which stopped auto-generating long-lived `kubernetes.io/service-account-token` Secrets for every ServiceAccount. Before 1.24, the SA controller automatically created a non-expiring token Secret and the admission controller injected it into every pod. Post-1.24, tokens are only provisioned via the TokenRequest API and mounted by the kubelet as projected volumes — but only when the projected volume is explicitly declared in the pod spec. Pods migrated from 1.23 without this volume declaration have no token on disk.

Can I just set automountServiceAccountToken: true to fix this quickly?

Setting `automountServiceAccountToken: true` at the ServiceAccount or Pod level will cause the kubelet to auto-inject the default projected volume — this resolves the immediate error. However, this is not recommended for production workloads because it grants API access to every container in the pod regardless of need, and you lose explicit control over token audience and expiry. The correct fix is to keep `automountServiceAccountToken: false` and declare an explicit projected volume with a scoped audience and a short `expirationSeconds` value.

How do I configure the projected SA token for Vault Agent or external OIDC audiences?

Set the `audience` field in the `serviceAccountToken` projected source to match your Vault JWT auth backend's `bound_audiences` value or your OIDC provider's expected audience string. For example, for HashiCorp Vault: `audience: https://vault.example.com`. The default value (`https://kubernetes.default.svc.cluster.local`) is only valid for in-cluster API server authentication. Using the wrong audience will result in a JWT validation failure at the target service, not a mount error.

Fixing 'Service Account Token Is Not Mounted' in Kubernetes 1.24+ with Projected Volumes

Threat/Impact Level: HIGH | Exploitability/Downtime Risk: HIGH | Time to Fix: 10 mins

TL;DR

What broke: Kubernetes 1.24 removed auto-generation of long-lived ServiceAccount secret tokens. Pods with automountServiceAccountToken: false or workloads migrated from pre-1.24 clusters have no token volume, causing any in-cluster API call to fail immediately with service account token is not mounted.
How to fix it: Explicitly define a projected volume with a serviceAccountToken source (and matching volumeMount) on every container that needs API server access. For workloads that need zero API access, confirm automountServiceAccountToken: false is intentional and remove the SDK calls.
Fast path: Use our Client-Side Sandbox below to auto-refactor your broken Pod/Deployment YAML without sending your configs to a third-party server.

The Incident (What Does the Error Mean?)

Raw error — typically surfaced in pod logs or the client-go / controller-runtime stack:

failed to create client: service account token is not mounted

or from the API server audit log:

Unable to authenticate the request due to an error:
[invalid bearer token, service account token has been invalidated]

Immediate consequence: Any pod using client-go, kubectl inside a container, Prometheus, Argo Workflows, Vault Agent, or any controller that calls rest.InClusterConfig() will crash-loop or hard-fail at startup. The path /var/run/secrets/kubernetes.io/serviceaccount/token does not exist on disk.

Why it changed: Prior to 1.24, the SA controller auto-created a kubernetes.io/service-account-token Secret and injected it. KEP-2799 / KEP-1205 removed this. Tokens are now exclusively provisioned via the TokenRequest API and mounted as short-lived projected volumes by the kubelet — but only if the projected volume is declared.

The Attack Vector / Blast Radius

This is a dual-risk finding — it is simultaneously a availability failure and a historical security debt item.

Availability blast radius:

Every workload relying on rest.InClusterConfig() fails. In a large cluster migration from 1.23 → 1.24, this can take down entire control-plane adjacent workloads (Prometheus, cert-manager, external-secrets-operator) simultaneously.
Helm chart releases that shipped pre-1.24 SA patterns will fail silently on upgrade if the chart author did not pin the projected volume spec.

Security angle (why the old pattern was dangerous):

Legacy kubernetes.io/service-account-token Secrets were non-expiring. A single leaked token (via a misconfigured kubectl get secret -o yaml in CI logs, a Velero backup in a public S3 bucket, or a compromised etcd snapshot) gave permanent API access.
The new TokenRequest-based projected tokens expire (default 1 hour, configurable via expirationSeconds). An attacker who exfiltrates the token from /var/run/secrets/ has a hard TTL window.
If your cluster still has legacy SA Secrets lingering from pre-1.24 (check: kubectl get secrets -A | grep kubernetes.io/service-account-token), those are live credentials with no expiry. Rotate them immediately.

How to Fix It

Basic Fix — Add the Projected Volume

apiVersion: v1
kind: Pod
metadata:
  name: my-controller
spec:
  serviceAccountName: my-sa
- # automountServiceAccountToken: false  <-- was set, or cluster default changed
+ automountServiceAccountToken: false   # keep false; we manage the mount explicitly
  containers:
  - name: app
    image: my-controller:latest
+   volumeMounts:
+   - name: kube-api-access
+     mountPath: /var/run/secrets/kubernetes.io/serviceaccount
+     readOnly: true
+ volumes:
+ - name: kube-api-access
+   projected:
+     defaultMode: 0444
+     sources:
+     - serviceAccountToken:
+         path: token
+         expirationSeconds: 3607
+         audience: https://kubernetes.default.svc.cluster.local
+     - configMap:
+         name: kube-root-ca.crt
+         items:
+         - key: ca.crt
+           path: ca.crt
+     - downwardAPI:
+         items:
+         - path: namespace
+           fieldRef:
+             apiVersion: v1
+             fieldPath: metadata.namespace

Critical: The mountPath must be exactly /var/run/secrets/kubernetes.io/serviceaccount. client-go's rest.InClusterConfig() hardcodes this path. Deviation requires custom token path configuration in the SDK.

Enterprise Best Practice — Least-Privilege SA + Bound Token

# 1. Dedicated, least-privilege ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-controller-sa
  namespace: production
- automountServiceAccountToken: true   # old default, dangerous
+ automountServiceAccountToken: false  # explicit opt-in per pod only
---
# 2. RBAC: grant only what the workload needs
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: my-controller-role
  namespace: production
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list", "watch"]
- # DO NOT add: ["secrets"] unless absolutely required
---
# 3. Pod spec with explicit projected volume
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      serviceAccountName: my-controller-sa
+     automountServiceAccountToken: false
      containers:
      - name: app
+       volumeMounts:
+       - name: kube-api-access-custom
+         mountPath: /var/run/secrets/kubernetes.io/serviceaccount
+         readOnly: true
+     volumes:
+     - name: kube-api-access-custom
+       projected:
+         defaultMode: 0444
+         sources:
+         - serviceAccountToken:
+             path: token
+             expirationSeconds: 3607
+             audience: https://kubernetes.default.svc.cluster.local
+         - configMap:
+             name: kube-root-ca.crt
+             items:
+             - key: ca.crt
+               path: ca.crt
+         - downwardAPI:
+             items:
+             - path: namespace
+               fieldRef:
+                 apiVersion: v1
+                 fieldPath: metadata.namespace

For Vault Agent / IRSA-style workloads: set audience to your Vault JWT auth backend audience string, not the default API server URL.

💡 Tired of pasting proprietary configs into ChatGPT? Generic AI tools log your company's ARNs, DB strings, and private keys. StackEngine is a zero-backend, pure Client-Side WASM utility. Drop your failing config into the sandbox above. We redact your secrets locally in the browser and auto-generate the refactored code using your own API key.

Prevention in CI/CD

1. OPA / Gatekeeper Policy — Deny Pods Without Projected SA Token

package k8s.projected_sa_token

violation[{"msg": msg}] {
  pod := input.review.object
  pod.kind == "Pod"
  not pod.spec.automountServiceAccountToken == false
  msg := "Pod must set automountServiceAccountToken: false and declare an explicit projected SA token volume."
}

violation[{"msg": msg}] {
  pod := input.review.object
  pod.kind == "Pod"
  pod.spec.automountServiceAccountToken == false
  volumes := {v.name | v := pod.spec.volumes[_]; v.projected}
  mounts := {m.name | m := pod.spec.containers[_].volumeMounts[_]}
  count(volumes & mounts) == 0
  msg := "automountServiceAccountToken is false but no projected SA token volume is mounted."
}

2. Checkov — Scan IaC Before Apply

# Run against your Helm-rendered manifests
helm template my-release ./chart | checkov -d - --check CKV_K8S_41
# CKV_K8S_41: Ensure SA tokens are not auto-mounted where not required
# Add custom check for projected volume presence in your .checkov.yaml

3. CI Gate — `kubectl --dry-run=server`

# .github/workflows/k8s-validate.yaml
- name: Server-side dry-run against staging cluster
  run: |
    helm template . | kubectl apply --dry-run=server \
      --validate=true \
      -f - \
      --server https://$STAGING_API_SERVER

Server-side dry-run will catch missing volume mounts that client-side schema validation misses.

4. Audit Existing Clusters for Legacy SA Secrets

# Find all non-expiring legacy SA tokens still in cluster
kubectl get secrets -A \
  -o jsonpath='{range .items[?(@.type=="kubernetes.io/service-account-token")]}{.metadata.namespace}{"\t"}{.metadata.name}{"\n"}{end}'

# For each found: verify if still referenced, then delete or annotate with expiry

Set a recurring audit job (weekly minimum) on production clusters. Legacy tokens from pre-1.24 workloads survive cluster upgrades and represent permanent credential exposure.