Fixing Containerd CRI 'Failed to Create Pod Sandbox' Caused by Image Pull Secret Mismatch
Threat/Impact Level: HIGH | Downtime Risk: HIGH | Time to Fix: 10–20 mins
TL;DR
- What broke: Containerd's CRI shim cannot pull the pause/sandbox image (or the app image) because the
imagePullSecretreferenced in the Pod spec either doesn't exist in the correct namespace, has a malformed.dockerconfigjson, or references the wrong registry host. - How to fix it: Verify the Secret exists in the same namespace as the Pod, decode and validate the
.dockerconfigjsonpayload, and ensure the registry hostname in the secret matches the image reference exactly. - Fast path: Use our Client-Side Sandbox above to auto-refactor your failing Pod spec and Secret manifest — paste both, get corrected YAML output instantly.
▼ 🔍 Root Cause Breakdown & Kernel Diagnostics [Click to Expand Deep Dive]
The Incident (What Does the Error Mean?)
You'll see this in kubectl describe pod <pod-name> or in the kubelet journal:
Warning Failed 3s kubelet Failed to create pod sandbox:
rpc error: code = Unknown
desc = failed to pull image "registry.internal.corp/pause:3.9":
failed to pull and unpack image "registry.internal.corp/pause:3.9":
failed to resolve reference "registry.internal.corp/pause:3.9":
unexpected status code 401 Unauthorizedor:
Warning Failed 2s kubelet Failed to create pod sandbox:
rpc error: code = Unknown
desc = failed to get sandbox image "registry.internal.corp/pause:3.9":
error getting credentials - err: docker-credential-ecr-login:
resolving credentials: secret "regcred" not foundImmediate consequence: The Pod never leaves ContainerCreating. No init containers run. No app containers start. If this is a Deployment rollout, the new ReplicaSet stalls, and depending on your maxUnavailable setting, the old pods may have already been terminated — you're now at zero healthy replicas.
▼ 🔍 Detailed Impact & Blast Radius Analysis [Click to Expand Deep Dive]
The Attack Vector / Blast Radius
This failure mode has two distinct blast radii:
1. Operational (most common): A namespace-scoped Secret was created in default but the workload runs in production. Containerd's CRI plugin calls the kubelet credential provider, which calls the Kubernetes API for the secret — it's not found, auth fails, sandbox never initializes. Every pod in that Deployment fails simultaneously if the rollout is already in progress.
2. Security regression (the dangerous one): Teams under pressure "fix" this by switching private images to public mirrors or by granting imagePullSecrets with a service account that has wildcard registry access. A misconfigured dockerconfigjson with "auths": {"https://index.docker.io/v1/": {}} (empty credentials) silently falls back to unauthenticated pulls on some runtimes — meaning your image supply chain is now unverified. Worse, if the secret contains credentials for *.amazonaws.com ECR and is bound to a service account with overly broad RBAC, a compromised pod can enumerate and pull any image in that registry account.
Cascading risk: HPA-triggered scale-out events will repeatedly attempt and fail sandbox creation, generating thundering-herd kubelet log spam and elevated API server load from credential resolution retries.
How to Fix It
Step 1 — Confirm the secret exists in the right namespace
kubectl get secret regcred -n production
# If NotFound — that's your problem.
# Check what namespace it actually lives in:
kubectl get secrets --all-namespaces | grep regcredStep 2 — Decode and validate the dockerconfigjson
kubectl get secret regcred -n production \
-o jsonpath='{.data.'\''.dockerconfigjson'\''}' | base64 -d | jq .You must see the exact registry hostname that matches your image reference:
{
"auths": {
"registry.internal.corp": {
"username": "svc-k8s-pull",
"password": "<token>",
"auth": "<base64(user:pass)>"
}
}
}If your image is registry.internal.corp/pause:3.9 but the secret has https://registry.internal.corp/v2/ — that mismatch is the bug. Containerd does exact-prefix matching on registry hosts.
Basic Fix — Recreate the secret with the correct hostname
kubectl create secret docker-registry regcred \
--docker-server=registry.internal.corp \
--docker-username=svc-k8s-pull \
--docker-password="$(cat /vault/secrets/registry-token)" \
-n production \
--dry-run=client -o yaml | kubectl apply -f -Enterprise Best Practice — Attach the secret to the namespace's default ServiceAccount
Instead of adding imagePullSecrets to every Pod spec (which gets missed), patch it onto the ServiceAccount so all pods in the namespace inherit it automatically:
apiVersion: v1
kind: ServiceAccount
metadata:
name: default
namespace: production
+imagePullSecrets:
+- name: regcredPod spec correction:
apiVersion: v1
kind: Pod
metadata:
name: app-worker
namespace: production
spec:
+ imagePullSecrets:
+ - name: regcred
containers:
- name: app
image: registry.internal.corp/app:v2.1.0
- # imagePullSecrets was missing or pointed to wrong secret nameSecret manifest (correct form):
apiVersion: v1
kind: Secret
metadata:
- name: reg-cred # wrong name — Pod spec referenced 'regcred'
+ name: regcred
- namespace: default # wrong namespace
+ namespace: production
type: kubernetes.io/dockerconfigjson
data:
- .dockerconfigjson: eyJhdXRocyI6eyJodHRwczovL3JlZ2lzdHJ5LmludGVybmFsLmNvcnAvdjIvIjp7fX19
- # ^ decoded: registry URL has https:// prefix + /v2/ suffix — containerd won't match this
+ .dockerconfigjson: eyJhdXRocyI6eyJyZWdpc3RyeS5pbnRlcm5hbC5jb3JwIjp7InVzZXJuYW1lIjoic3ZjLWs4cy1wdWxsIiwicGFzc3dvcmQiOiJ0b2tlbiIsImF1dGgiOiJiYXNlNjRlbmNvZGVkIn19fQ==
+ # ^ decoded: bare hostname 'registry.internal.corp' — correct for containerd💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.
(For CI/CD pipeline automation, install via
npm i -g stackengine).
▼ 🔍 Continuous Pipeline Governance & CI/CD Guardrails [Click to Expand Deep Dive]
Prevention in CI/CD
1. OPA/Gatekeeper policy — enforce imagePullSecrets presence:
package k8srequiredimagepullsecrets
violation[{"msg": msg}] {
input.review.object.kind == "Pod"
count(input.review.object.spec.imagePullSecrets) == 0
not input.review.object.spec.serviceAccountName # no SA-level secret either
msg := sprintf("Pod '%v' in namespace '%v' has no imagePullSecrets",
[input.review.object.metadata.name, input.review.object.metadata.namespace])
}2. Checkov scan in your pipeline:
checkov -f pod.yaml --check CKV_K8S_35 # Ensures imagePullSecrets is set3. Helm chart values.yaml guard:
# In your chart's _helpers.tpl, fail fast at render time:
{{- if and .Values.image.registry (not .Values.imagePullSecrets) }}
{{- fail "imagePullSecrets must be set when using a private registry" }}
{{- end }}4. Namespace bootstrap automation: Use a Namespace provisioning controller (e.g., Hierarchical Namespace Controller or a simple operator) that automatically copies the regcred secret into every new namespace and patches the default ServiceAccount. Never rely on humans remembering to do this during incident-driven namespace creation.
5. Validate secret format in CI before deploy:
# In your GitHub Actions / GitLab CI pre-deploy step:
kubectl create secret docker-registry regcred \
--docker-server="$REGISTRY_HOST" \
--docker-username="$REGISTRY_USER" \
--docker-password="$REGISTRY_PASS" \
--dry-run=client -o json | \
jq -r '.data[".dockerconfigjson"]' | \
base64 -d | jq -e '.auths | keys[] | test("^[a-z0-9.-]+$")' \
|| (echo "FATAL: Registry hostname has invalid format for containerd" && exit 1)This regex rejects https:// prefixes and trailing slashes before the secret ever reaches the cluster.
Official Grounding & Normative Specifications
The technical diagnostics, error recovery protocols, and prevention patterns in this specification are grounded in official vendor documentation and standards:
Industry-standard specification for container execution boundaries and POSIX namespace isolation.
Federal security guidelines for secret exfiltration prevention, automated integrity checks, and peer sign-offs.
Core memory subsystem controls, pressure stall information (PSI), and kernel OOM invocation mechanics.