How to Fix CreateContainerConfigError: Secret Not Found in Kubernetes

Threat/Impact Level: HIGH | Exploitability/Downtime Risk: HIGH | Time to Fix: 10 mins

TL;DR

• What broke: Kubernetes cannot start your container because the pod spec references a Secret object that does not exist in the pod's namespace — the kubelet fails at config materialization before the container ever runs.
• How to fix it: Either create the missing Secret (kubectl create secret) or correct the secret name/namespace reference in your pod spec.
• Use our Client-Side Sandbox above to paste your failing pod YAML and auto-diagnose the broken secretKeyRef, secretRef, or volumes.secret reference.

The Incident (What Does the Error Mean?)

Raw event output from kubectl describe pod <pod-name>:

Warning  Failed     3s    kubelet  Error: CreateContainerConfigError
Message: secret "db-credentials" not found

The kubelet resolves all Secret and ConfigMap references before calling the container runtime. If any referenced Secret is absent, the pod is stuck in Pending with CreateContainerConfigError — it never reaches ContainerCreating. Every replica in the Deployment is dead. Your service is down.

The Attack Vector / Blast Radius

This is not just a misconfiguration annoyance. The blast radius:

• Full deployment outage. Every pod in the ReplicaSet fails identically. No rollback helps unless the previous revision also had the secret — and if you just rotated it, it probably didn't.
• Silent CI/CD gap. Most pipelines kubectl apply the Deployment manifest but never verify that dependent Secrets were applied first. A secret deleted by a namespace cleanup job, a Helm rollback, or a Vault sync failure silently kills prod on the next pod reschedule (node drain, HPA scale-up, spot termination).
• Secret sprawl risk. The reactive fix — kubectl create secret generic with a hardcoded value pasted from Slack — is how plaintext credentials end up in shell history, CI logs, and incident tickets. This is a credential hygiene incident waiting to happen.
• Namespace isolation failure. Secrets are namespace-scoped. A pod in namespace: production cannot read a secret in namespace: staging. Cross-namespace secret references silently fail here, and engineers waste 20 minutes before checking the namespace.

How to Fix It (The Solution)

Basic Fix — Create the Missing Secret

First, confirm exactly what's missing:

# Check what the pod expects
kubectl get pod <pod-name> -o jsonpath='{.spec.containers[*].env[*].valueFrom.secretKeyRef}'

# Check what exists in the namespace
kubectl get secrets -n <namespace>

Create the secret if it's genuinely absent:

kubectl create secret generic db-credentials \
  --from-literal=DB_PASSWORD='your-password' \
  --from-literal=DB_USER='your-user' \
  -n production

Enterprise Best Practice — Declarative Secret Management with Correct Reference Hygiene

The pod spec was likely referencing a wrong name or the secret was never provisioned by the pipeline. Fix both the reference and the provisioning:

# pod-spec.yaml
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: api-server
   namespace: production
 spec:
   template:
     spec:
       containers:
       - name: api
         image: myapp:1.4.2
         env:
-        - name: DB_PASSWORD
-          valueFrom:
-            secretKeyRef:
-              name: database-creds        # secret didn't exist
-              key: password
+        - name: DB_PASSWORD
+          valueFrom:
+            secretKeyRef:
+              name: db-credentials        # matches the actual Secret object name
+              key: DB_PASSWORD
+              optional: false             # explicit — fail fast, don't silently inject empty string

For Vault or AWS Secrets Manager backed secrets, use the External Secrets Operator so the ExternalSecret CR creates the native Kubernetes Secret before any pod references it:

-# Manual kubectl create secret (breaks in GitOps, leaks to shell history)
-kubectl create secret generic db-credentials --from-literal=DB_PASSWORD=$PASS

+# ExternalSecret CR — declarative, auditable, rotation-aware
+apiVersion: external-secrets.io/v1beta1
+kind: ExternalSecret
+metadata:
+  name: db-credentials
+  namespace: production
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    name: vault-backend
+    kind: ClusterSecretStore
+  target:
+    name: db-credentials          # this is what the pod references
+    creationPolicy: Owner
+  data:
+  - secretKey: DB_PASSWORD
+    remoteRef:
+      key: secret/production/database
+      property: password

💡 Tired of pasting proprietary configs into ChatGPT? Generic AI tools log your company's ARNs, DB strings, and private keys. StackEngine is a zero-backend, pure Client-Side WASM utility. Drop your failing config into the sandbox above. We redact your secrets locally in the browser and auto-generate the refactored code using your own API key.

Prevention in CI/CD

1. Helm / Kustomize pre-flight validation

Add a helm lint + kubectl apply --dry-run=server step. Server-side dry-run hits the API server and will reject a pod spec referencing a non-existent secret — catching this in the pipeline, not in prod.

# In your GitHub Actions / GitLab CI step
kubectl apply --dry-run=server -f k8s/deployment.yaml

2. OPA/Gatekeeper policy — enforce secret pre-existence

# ConstraintTemplate: require referenced secrets to exist before pod admission
violation[{"msg": msg}] {
  container := input.review.object.spec.containers[_]
  ref := container.env[_].valueFrom.secretKeyRef.name
  not data.inventory.namespace[input.review.object.metadata.namespace]["v1"]["Secret"][ref]
  msg := sprintf("Secret '%v' referenced in pod does not exist in namespace '%v'", [ref, input.review.object.metadata.namespace])
}

3. Checkov static scan on manifests

checkov -f k8s/deployment.yaml --check CKV_K8S_35

CKV_K8S_35 flags pods that do not use secrets as files or env vars from a verified source — catches dangling references in static analysis before kubectl apply.

4. Deployment ordering in GitOps (ArgoCD / Flux)

Use ArgoCD syncWave annotations to guarantee Secrets (or ExternalSecrets) are applied and healthy before Deployments:

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"   # Secrets wave runs first

Deployments default to wave 0 — they will not start until wave -1 resources are Healthy.