RAPID INCIDENT TRIAGE & ACTION PLAN•Security
01. Error Signature
K8s 1.24+ SA Token Projection Verifier
02. Root Cause Vector

Kubernetes 1.24+ removed auto-mounted legacy SA tokens; pods without explicit projected volume mounts fail API server auth with 'service ...

03. Rapid Prescription

Inspect container exit codes via kubectl describe, verify entrypoint and readiness probes.

Zero-Trust Client IDE/ Security

K8s 1.24+ SA Token Projection Verifier

Kubernetes 1.24+ removed auto-mounted legacy SA tokens; pods without explicit projected volume mounts fail API server auth with 'service account token is not mounted'.

SYSTEM STATUS
CREDITS: 0 / 3
source.yaml(69L)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
apiVersion: v1
kind: Pod
metadata:
  name: legacy-payment-processor-v2
  namespace: pci-secure-zone
  labels:
    app: payment-processor
    version: legacy-v2
    # TODO: migrate to new service mesh after Q4
    # owner: team-argon@my-registry.internal
spec:
  # OLD config kept for rollback - do not remove
  # serviceAccountName: default
  automountServiceAccountToken: false  # Security team mandate for PCI pods
  containers:
  - name: payment-api
    image: my-registry.internal/legacy-api:stable-2.1.4
    # VULNERABILITY: Projected volume mount missing service account token path due to automountServiceAccountToken conflict
    volumeMounts:
    - name: token-vol
      mountPath: /var/run/secrets/tokens
      readOnly: true
    - name: config-vol
      mountPath: /etc/app/config
    env:
    - name: AWS_ACCESS_KEY_ID
      value: "AKIAIOSFODNN7EXAMPLE"
    - name: AWS_SECRET_ACCESS_KEY
      value: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
    - name: STRIPE_API_KEY
      value: "sk_live_4eC39HqLyjWDarjtT1zdp7dc"
    - name: DB_CONNECTION_STRING
      value: "postgresql://user:SuperSecretPass123!@legacy-db.pci-secure-zone.svc.cluster.local:5432/payments?sslmode=require"
    # TODO: replace with vault sidecar injection
    resources:
      requests:
        memory: "256Mi"
        cpu: "250m"
      limits:
        memory: "512Mi"
        cpu: "500m"
  volumes:
  - name: token-vol
    projected:
      sources:
      - configMap:
          name: app-config
      - downwardAPI:
          items:
          - path: "labels"
            fieldRef:
              fieldPath: metadata.labels
      # MISSING: serviceAccountToken source - token won't be mounted in 1.24+
      # - serviceAccountToken:
      #     path: token
      #     expirationSeconds: 3600
  - name: config-vol
    configMap:
      name: payment-config
  # Legacy security context from old compliance template
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000
    # seLinuxOptions:
    #   level: "s0:c123,c456"
  # nodeSelector:
  #   zone: pci-secure
AI Audit Console
> Waiting for pre-flight audit...
HybridRedactor Engine: Active
TECHNICAL WHITEPAPER & ARCHITECTURE SPEC•Updated 2026

Fixing 'Service Account Token Is Not Mounted' in Kubernetes 1.24+ with Projected Volumes

Threat/Impact Level: HIGH | Exploitability/Downtime Risk: HIGH | Time to Fix: 10 mins


TL;DR

  • What broke: Kubernetes 1.24 removed auto-generation of long-lived ServiceAccount secret tokens. Pods with automountServiceAccountToken: false or workloads migrated from pre-1.24 clusters have no token volume, causing any in-cluster API call to fail immediately with service account token is not mounted.
  • How to fix it: Explicitly define a projected volume with a serviceAccountToken source (and matching volumeMount) on every container that needs API server access. For workloads that need zero API access, confirm automountServiceAccountToken: false is intentional and remove the SDK calls.
  • Fast path: Use our Client-Side Sandbox above to auto-refactor your broken Pod/Deployment YAML without sending your configs to a third-party server.

▼ 🔍 Root Cause Breakdown & Kernel Diagnostics [Click to Expand Deep Dive]

The Incident (What Does the Error Mean?)

Raw error — typically surfaced in pod logs or the client-go / controller-runtime stack:

failed to create client: service account token is not mounted

or from the API server audit log:

Unable to authenticate the request due to an error:
[invalid bearer token, service account token has been invalidated]

Immediate consequence: Any pod using client-go, kubectl inside a container, Prometheus, Argo Workflows, Vault Agent, or any controller that calls rest.InClusterConfig() will crash-loop or hard-fail at startup. The path /var/run/secrets/kubernetes.io/serviceaccount/token does not exist on disk.

Why it changed: Prior to 1.24, the SA controller auto-created a kubernetes.io/service-account-token Secret and injected it. KEP-2799 / KEP-1205 removed this. Tokens are now exclusively provisioned via the TokenRequest API and mounted as short-lived projected volumes by the kubelet — but only if the projected volume is declared.


▼ 🔍 Detailed Impact & Blast Radius Analysis [Click to Expand Deep Dive]

The Attack Vector / Blast Radius

This is a dual-risk finding — it is simultaneously a availability failure and a historical security debt item.

Availability blast radius:

  • Every workload relying on rest.InClusterConfig() fails. In a large cluster migration from 1.23 → 1.24, this can take down entire control-plane adjacent workloads (Prometheus, cert-manager, external-secrets-operator) simultaneously.
  • Helm chart releases that shipped pre-1.24 SA patterns will fail silently on upgrade if the chart author did not pin the projected volume spec.

Security angle (why the old pattern was dangerous):

  • Legacy kubernetes.io/service-account-token Secrets were non-expiring. A single leaked token (via a misconfigured kubectl get secret -o yaml in CI logs, a Velero backup in a public S3 bucket, or a compromised etcd snapshot) gave permanent API access.
  • The new TokenRequest-based projected tokens expire (default 1 hour, configurable via expirationSeconds). An attacker who exfiltrates the token from /var/run/secrets/ has a hard TTL window.
  • If your cluster still has legacy SA Secrets lingering from pre-1.24 (check: kubectl get secrets -A | grep kubernetes.io/service-account-token), those are live credentials with no expiry. Rotate them immediately.

How to Fix It

Basic Fix — Add the Projected Volume

apiVersion: v1
kind: Pod
metadata:
  name: my-controller
spec:
  serviceAccountName: my-sa
- # automountServiceAccountToken: false  <-- was set, or cluster default changed
+ automountServiceAccountToken: false   # keep false; we manage the mount explicitly
  containers:
  - name: app
    image: my-controller:latest
+   volumeMounts:
+   - name: kube-api-access
+     mountPath: /var/run/secrets/kubernetes.io/serviceaccount
+     readOnly: true
+ volumes:
+ - name: kube-api-access
+   projected:
+     defaultMode: 0444
+     sources:
+     - serviceAccountToken:
+         path: token
+         expirationSeconds: 3607
+         audience: https://kubernetes.default.svc.cluster.local
+     - configMap:
+         name: kube-root-ca.crt
+         items:
+         - key: ca.crt
+           path: ca.crt
+     - downwardAPI:
+         items:
+         - path: namespace
+           fieldRef:
+             apiVersion: v1
+             fieldPath: metadata.namespace

Critical: The mountPath must be exactly /var/run/secrets/kubernetes.io/serviceaccount. client-go's rest.InClusterConfig() hardcodes this path. Deviation requires custom token path configuration in the SDK.


Enterprise Best Practice — Least-Privilege SA + Bound Token

# 1. Dedicated, least-privilege ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-controller-sa
  namespace: production
- automountServiceAccountToken: true   # old default, dangerous
+ automountServiceAccountToken: false  # explicit opt-in per pod only
---
# 2. RBAC: grant only what the workload needs
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: my-controller-role
  namespace: production
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list", "watch"]
- # DO NOT add: ["secrets"] unless absolutely required
---
# 3. Pod spec with explicit projected volume
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      serviceAccountName: my-controller-sa
+     automountServiceAccountToken: false
      containers:
      - name: app
+       volumeMounts:
+       - name: kube-api-access-custom
+         mountPath: /var/run/secrets/kubernetes.io/serviceaccount
+         readOnly: true
+     volumes:
+     - name: kube-api-access-custom
+       projected:
+         defaultMode: 0444
+         sources:
+         - serviceAccountToken:
+             path: token
+             expirationSeconds: 3607
+             audience: https://kubernetes.default.svc.cluster.local
+         - configMap:
+             name: kube-root-ca.crt
+             items:
+             - key: ca.crt
+               path: ca.crt
+         - downwardAPI:
+             items:
+             - path: namespace
+               fieldRef:
+                 apiVersion: v1
+                 fieldPath: metadata.namespace

For Vault Agent / IRSA-style workloads: set audience to your Vault JWT auth backend audience string, not the default API server URL.


💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.

(For CI/CD pipeline automation, install via npm i -g stackengine).


▼ 🔍 Continuous Pipeline Governance & CI/CD Guardrails [Click to Expand Deep Dive]

Prevention in CI/CD

1. OPA / Gatekeeper Policy — Deny Pods Without Projected SA Token

package k8s.projected_sa_token

violation[{"msg": msg}] {
  pod := input.review.object
  pod.kind == "Pod"
  not pod.spec.automountServiceAccountToken == false
  msg := "Pod must set automountServiceAccountToken: false and declare an explicit projected SA token volume."
}

violation[{"msg": msg}] {
  pod := input.review.object
  pod.kind == "Pod"
  pod.spec.automountServiceAccountToken == false
  volumes := {v.name | v := pod.spec.volumes[_]; v.projected}
  mounts := {m.name | m := pod.spec.containers[_].volumeMounts[_]}
  count(volumes & mounts) == 0
  msg := "automountServiceAccountToken is false but no projected SA token volume is mounted."
}

2. Checkov — Scan IaC Before Apply

# Run against your Helm-rendered manifests
helm template my-release ./chart | checkov -d - --check CKV_K8S_41
# CKV_K8S_41: Ensure SA tokens are not auto-mounted where not required
# Add custom check for projected volume presence in your .checkov.yaml

3. CI Gate — `kubectl --dry-run=server`

# .github/workflows/k8s-validate.yaml
- name: Server-side dry-run against staging cluster
  run: |
    helm template . | kubectl apply --dry-run=server \
      --validate=true \
      -f - \
      --server https://$STAGING_API_SERVER

Server-side dry-run will catch missing volume mounts that client-side schema validation misses.

4. Audit Existing Clusters for Legacy SA Secrets

# Find all non-expiring legacy SA tokens still in cluster
kubectl get secrets -A \
  -o jsonpath='{range .items[?(@.type=="kubernetes.io/service-account-token")]}{.metadata.namespace}{"\t"}{.metadata.name}{"\n"}{end}'

# For each found: verify if still referenced, then delete or annotate with expiry

Set a recurring audit job (weekly minimum) on production clusters. Legacy tokens from pre-1.24 workloads survive cluster upgrades and represent permanent credential exposure.

Official Grounding & Normative Specifications

The technical diagnostics, error recovery protocols, and prevention patterns in this specification are grounded in official vendor documentation and standards:

Diagnostic & Architectural FAQ

Q1.Why did my pods work on Kubernetes 1.23 but break after upgrading to 1.24+?
Kubernetes 1.24 implemented KEP-2799, which stopped auto-generating long-lived `kubernetes.io/service-account-token` Secrets for every ServiceAccount. Before 1.24, the SA controller automatically created a non-expiring token Secret and the admission controller injected it into every pod. Post-1.24, tokens are only provisioned via the TokenRequest API and mounted by the kubelet as projected volumes — but only when the projected volume is explicitly declared in the pod spec. Pods migrated from 1.23 without this volume declaration have no token on disk. This is a fundamental architectural shift that demands precise, declarative security posture.
Q2.Can I just set automountServiceAccountToken: true to fix this quickly?
Setting `automountServiceAccountToken: true` at the ServiceAccount or Pod level will cause the kubelet to auto-inject the default projected volume — this resolves the immediate error. However, this is a critical security vulnerability. It grants unrestricted API access to every container in the pod, creating a massive blast radius. You lose all control over token audience and expiry, violating zero-trust principles. Manual fixes are error-prone and risk misconfiguration. Instead, drop the broken manifest into the StackEngine Local Sandbox above to automatically compile a zero-trust fix with scoped audience and short expirationSeconds.
Q3.How do I configure the projected SA token for Vault Agent or external OIDC audiences?
Set the `audience` field in the `serviceAccountToken` projected source to match your Vault JWT auth backend's `bound_audiences` value or your OIDC provider's expected audience string. For example, for HashiCorp Vault: `audience: https://vault.example.com`. The default value (`https://kubernetes.default.svc.cluster.local`) is only valid for in-cluster API server authentication. Using the wrong audience will result in a JWT validation failure at the target service, not a mount error. Doing this manually across large mono-repos or at scale is a compliance nightmare. Deploy the StackEngine CI/CD Automator (Team Edition) for on-prem, automated pull request generation with Maker-Checker review and policy enforcement.