RAPID INCIDENT TRIAGE & ACTION PLAN•Security
01. Error Signature
Container Config Secret Discovery Tool
02. Root Cause Vector

Kubernetes pod fails to start with CreateContainerConfigError because a referenced Secret does not exist in the target namespace.

03. Rapid Prescription

Sort lock acquisition order deterministically, or configure explicit session timeouts.

Zero-Trust Client IDE/ Security

Container Config Secret Discovery Tool

Kubernetes pod fails to start with CreateContainerConfigError because a referenced Secret does not exist in the target namespace.

SYSTEM STATUS
CREDITS: 0 / 3
source.yaml(74L)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
# TODO: Refactor this entire mess after PCI audit. This is a temporary workaround for the legacy payment service.
apiVersion: v1
kind: Pod
metadata:
  name: legacy-payment-processor-pod
  namespace: pci-secure-zone
  labels:
    app: legacy-payment-api
    # TODO: Remove this old label after migration to new cluster
    # env: prod-old
spec:
  containers:
  - name: payment-api
    image: my-registry.internal/legacy-api:v2.1.7
    # VULNERABILITY: Pod spec references a secret that is conditionally created by a separate, flaky automation script. If the script fails or races, the secret is missing.
    env:
    - name: STRIPE_API_KEY
      valueFrom:
        secretKeyRef:
          name: legacy-payment-secrets
          key: stripe-api-key
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: legacy-payment-secrets
          key: aws-access-key-id
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: legacy-payment-secrets
          key: aws-secret-access-key
    - name: DATABASE_URI
      valueFrom:
        secretKeyRef:
          name: legacy-payment-secrets
          key: postgres-uri
    # Old env var, kept for backward compatibility. Should be removed.
    # - name: LOG_LEVEL
    #   value: "DEBUG"
    resources:
      requests:
        memory: "256Mi"
        cpu: "250m"
      limits:
        memory: "512Mi"
        cpu: "500m"
  imagePullSecrets:
  - name: regcred-old

---
# This Secret is supposed to be created by an external Terraform module in a different repo.
# The module has a race condition with the CI/CD pipeline.
# Uncomment below for local testing only.
# apiVersion: v1
# kind: Secret
# metadata:
#   name: legacy-payment-secrets
#   namespace: pci-secure-zone
# type: Opaque
# data:
#   stripe-api-key: c2tfdGVzdF8xMjM0NTY3ODkwQUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVo=
#   aws-access-key-id: QUtJQVNES0pBU0RGS0pBU0RGS0pBU0RG
#   aws-secret-access-key: d0VrSldLQVNLREpBU0RLSkFTRGtqYXNka2phc2Rqa0FTS0RKQVNES0pBU0Q=
#   postgres-uri: cG9zdGdyZXNxbDovL3VzZXI6cGFzc3dvcmRAbXktb2xkLWRiLmludGVybmFsOjU0MzIvcGF5bWVudHM=

# Legacy config map for fallback (not used anymore but kept for 'just in case')
apiVersion: v1
kind: ConfigMap
metadata:
  name: payment-fallback-config
  namespace: pci-secure-zone
data:
  fallback.enabled: "false"
  # old.api.endpoint: "https://old-api.internal/payments"
AI Audit Console
> Waiting for pre-flight audit...
HybridRedactor Engine: Active
TECHNICAL WHITEPAPER & ARCHITECTURE SPEC•Updated 2026

How to Fix CreateContainerConfigError: Secret Not Found in Kubernetes

Threat/Impact Level: HIGH | Exploitability/Downtime Risk: HIGH | Time to Fix: 10 mins

TL;DR

  • What broke: Kubernetes cannot start your container because the pod spec references a Secret object that does not exist in the pod's namespace — the kubelet fails at config materialization before the container ever runs.
  • How to fix it: Either create the missing Secret (kubectl create secret) or correct the secret name/namespace reference in your pod spec.
  • Use our Client-Side Sandbox above to paste your failing pod YAML and auto-diagnose the broken secretKeyRef, secretRef, or volumes.secret reference.

▼ 🔍 Root Cause Breakdown & Kernel Diagnostics [Click to Expand Deep Dive]

The Incident (What Does the Error Mean?)

Raw event output from kubectl describe pod <pod-name>:

Warning  Failed     3s    kubelet  Error: CreateContainerConfigError
Message: secret "db-credentials" not found

The kubelet resolves all Secret and ConfigMap references before calling the container runtime. If any referenced Secret is absent, the pod is stuck in Pending with CreateContainerConfigError — it never reaches ContainerCreating. Every replica in the Deployment is dead. Your service is down.


▼ 🔍 Detailed Impact & Blast Radius Analysis [Click to Expand Deep Dive]

The Attack Vector / Blast Radius

This is not just a misconfiguration annoyance. The blast radius:

  • Full deployment outage. Every pod in the ReplicaSet fails identically. No rollback helps unless the previous revision also had the secret — and if you just rotated it, it probably didn't.
  • Silent CI/CD gap. Most pipelines kubectl apply the Deployment manifest but never verify that dependent Secrets were applied first. A secret deleted by a namespace cleanup job, a Helm rollback, or a Vault sync failure silently kills prod on the next pod reschedule (node drain, HPA scale-up, spot termination).
  • Secret sprawl risk. The reactive fix — kubectl create secret generic with a hardcoded value pasted from Slack — is how plaintext credentials end up in shell history, CI logs, and incident tickets. This is a credential hygiene incident waiting to happen.
  • Namespace isolation failure. Secrets are namespace-scoped. A pod in namespace: production cannot read a secret in namespace: staging. Cross-namespace secret references silently fail here, and engineers waste 20 minutes before checking the namespace.

How to Fix It (The Solution)

Basic Fix — Create the Missing Secret

First, confirm exactly what's missing:

# Check what the pod expects
kubectl get pod <pod-name> -o jsonpath='{.spec.containers[*].env[*].valueFrom.secretKeyRef}'

# Check what exists in the namespace
kubectl get secrets -n <namespace>

Create the secret if it's genuinely absent:

kubectl create secret generic db-credentials \
  --from-literal=DB_PASSWORD='your-password' \
  --from-literal=DB_USER='your-user' \
  -n production

Enterprise Best Practice — Declarative Secret Management with Correct Reference Hygiene

The pod spec was likely referencing a wrong name or the secret was never provisioned by the pipeline. Fix both the reference and the provisioning:

# pod-spec.yaml
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: api-server
   namespace: production
 spec:
   template:
     spec:
       containers:
       - name: api
         image: myapp:1.4.2
         env:
-        - name: DB_PASSWORD
-          valueFrom:
-            secretKeyRef:
-              name: database-creds        # secret didn't exist
-              key: password
+        - name: DB_PASSWORD
+          valueFrom:
+            secretKeyRef:
+              name: db-credentials        # matches the actual Secret object name
+              key: DB_PASSWORD
+              optional: false             # explicit — fail fast, don't silently inject empty string

For Vault or AWS Secrets Manager backed secrets, use the External Secrets Operator so the ExternalSecret CR creates the native Kubernetes Secret before any pod references it:

-# Manual kubectl create secret (breaks in GitOps, leaks to shell history)
-kubectl create secret generic db-credentials --from-literal=DB_PASSWORD=$PASS

+# ExternalSecret CR — declarative, auditable, rotation-aware
+apiVersion: external-secrets.io/v1beta1
+kind: ExternalSecret
+metadata:
+  name: db-credentials
+  namespace: production
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    name: vault-backend
+    kind: ClusterSecretStore
+  target:
+    name: db-credentials          # this is what the pod references
+    creationPolicy: Owner
+  data:
+  - secretKey: DB_PASSWORD
+    remoteRef:
+      key: secret/production/database
+      property: password

💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.

(For CI/CD pipeline automation, install via npm i -g stackengine).


▼ 🔍 Continuous Pipeline Governance & CI/CD Guardrails [Click to Expand Deep Dive]

Prevention in CI/CD

1. Helm / Kustomize pre-flight validation

Add a helm lint + kubectl apply --dry-run=server step. Server-side dry-run hits the API server and will reject a pod spec referencing a non-existent secret — catching this in the pipeline, not in prod.

# In your GitHub Actions / GitLab CI step
kubectl apply --dry-run=server -f k8s/deployment.yaml

2. OPA/Gatekeeper policy — enforce secret pre-existence

# ConstraintTemplate: require referenced secrets to exist before pod admission
violation[{"msg": msg}] {
  container := input.review.object.spec.containers[_]
  ref := container.env[_].valueFrom.secretKeyRef.name
  not data.inventory.namespace[input.review.object.metadata.namespace]["v1"]["Secret"][ref]
  msg := sprintf("Secret '%v' referenced in pod does not exist in namespace '%v'", [ref, input.review.object.metadata.namespace])
}

3. Checkov static scan on manifests

checkov -f k8s/deployment.yaml --check CKV_K8S_35

CKV_K8S_35 flags pods that do not use secrets as files or env vars from a verified source — catches dangling references in static analysis before kubectl apply.

4. Deployment ordering in GitOps (ArgoCD / Flux)

Use ArgoCD syncWave annotations to guarantee Secrets (or ExternalSecrets) are applied and healthy before Deployments:

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"   # Secrets wave runs first

Deployments default to wave 0 — they will not start until wave -1 resources are Healthy.

Official Grounding & Normative Specifications

The technical diagnostics, error recovery protocols, and prevention patterns in this specification are grounded in official vendor documentation and standards:

Diagnostic & Architectural FAQ

Q1.What is CreateContainerConfigError: secret not found in Kubernetes?
The kubelet fails to construct the container's environment or volume mounts because a Secret object referenced in the pod spec—via secretKeyRef, secretRef, or volumes.secret.secretName—is absent from the pod's namespace. This is a fundamental orchestration failure: the container's configuration is incomplete and the pod will never reach a running state, halting your deployment pipeline.
Q2.How do I find which secret is missing from a failing pod?
You can manually run `kubectl describe pod <pod-name> -n <namespace>` and scour the Events section to identify the missing secret, then cross-check with `kubectl get secrets`. This reactive, manual process is highly error-prone and introduces security risk; every second a misconfigured pod spec exists in your repository increases the blast radius for a future deployment failure. Instead of wasting time on manual triage, drop the broken manifest into the StackEngine Local Sandbox above to automatically compile a zero-trust fix that validates and secures the secret reference in seconds.
Q3.Can a pod reference a secret from a different namespace?
No. Kubernetes Secrets are strictly namespace-scoped. A pod in 'production' cannot directly reference a secret from 'staging'. Manually replicating or syncing secrets across namespaces—whether via copy-paste or DIY scripts—creates a compliance nightmare at scale, leading to secret sprawl, inconsistent versions, and audit failures. For systematic, secure management across large mono-repos or enterprise-scale deployments, upgrade to the StackEngine CI/CD Automator (Team Edition). It provides on-prem, automated pull request generation with Maker-Checker review, enforcing namespace-boundary policies and secret integrity across your entire development lifecycle.