Helm Registry Manifest Error Locator

The Incident (What Does the Error Mean?)

This is what kubectl describe pod <pod-name> shows during the outage:

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  2m                 default-scheduler  Successfully assigned default/app-6d7f9b-xkq2p to node-1
  Normal   Pulling    2m                 kubelet            Pulling image "registry.gitlab.com/myorg/myapp/nginx:latest"
  Warning  Failed     112s               kubelet            Failed to pull image "registry.gitlab.com/myorg/myapp/nginx:latest": rpc error: code = NotFound desc = failed to pull and unpack image "registry.gitlab.com/myorg/myapp/nginx:latest": failed to resolve reference "registry.gitlab.com/myorg/myapp/nginx:latest": unexpected status code 401 Unauthorized
  Warning  Failed     112s               kubelet            Error: ErrImagePull
  Warning  BackOff    98s (x3 over 110s) kubelet            Back-off pulling image "registry.gitlab.com/myorg/myapp/nginx:latest"
  Warning  Failed     98s (x3 over 110s) kubelet            Error: ImagePullBackOff

Immediate consequence: The pod never reaches Running state. It oscillates between ErrImagePull and ImagePullBackOff. Every replica in the Deployment is dead. Your service is down.

There are three distinct failure modes that produce this identical error surface:

1. Tag does not exist — nginx:latest was never pushed to registry.gitlab.com/myorg/myapp/, or was deleted/overwritten and the manifest digest is gone.
2. imagePullSecret is absent or bound to the wrong namespace — The kubelet cannot authenticate to the private registry at all (401/403).
3. Helm chart image values are wrong — image.repository or image.tag in values.yaml points to a non-existent path or uses a stale override.

The Attack Vector / Blast Radius

This is not just an ops inconvenience. The blast radius is significant:

Deployment blast radius:

• All pods in the Deployment/StatefulSet/DaemonSet referencing this image are unschedulable. A rolling update leaves you with zero healthy replicas if the old ReplicaSet was already scaled down.
• Horizontal Pod Autoscaler scale-out events will also fail silently — new pods spin up into ImagePullBackOff under load.

Security angle — why latest is a liability:

• Using latest means your CI/CD pipeline can silently push a broken or malicious layer to the tag and every future pod restart pulls it automatically. There is no pinning, no auditability, no rollback path via image tag alone.
• In a supply chain attack scenario (e.g., a compromised GitLab CI runner), an attacker can overwrite latest with a backdoored image. Because imagePullPolicy: Always re-pulls on every pod restart, the malicious image propagates to all nodes on the next rollout or node failure recovery — without any Helm release change being made.
• A missing or overly-permissive imagePullSecret stored as a plain Kubernetes Secret with no RBAC restriction means any pod in the namespace can mount it and exfiltrate your GitLab registry credentials.

Prevention in CI/CD

Stop this from ever hitting production again. These controls are non-negotiable for any team past 5 engineers.

1. Enforce non-`latest` tags with OPA/Gatekeeper

Deploy this ConstraintTemplate to block any image referencing latest:

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8sdenylatesttag
spec:
  crd:
    spec:
      names:
        kind: K8sDenyLatestTag
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8sdenylatesttag
        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          endswith(container.image, ":latest")
          msg := sprintf("Container '%v' uses ':latest' tag. Pin to an immutable tag or digest.", [container.name])
        }
        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          not contains(container.image, ":")
          msg := sprintf("Container '%v' has no image tag. Explicit tag or digest required.", [container.name])
        }

2. Checkov scan in your GitLab CI pipeline

# .gitlab-ci.yml
checkov-scan:
  stage: validate
  image: bridgecrew/checkov:latest
  script:
    - checkov -d ./helm-chart/templates --check CKV_K8S_14,CKV_K8S_15
    # CKV_K8S_14: Image Tag should be fixed - not latest
    # CKV_K8S_15: Image should use Always pull policy (paired with digest pinning)
  allow_failure: false

3. Automate secret rotation with External Secrets Operator

Never manually kubectl create secret again. Use External Secrets Operator to sync your GitLab deploy token from HashiCorp Vault or AWS Secrets Manager directly into the namespace as a kubernetes.io/dockerconfigjson secret — with automatic rotation.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: gitlab-registry-secret
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: gitlab-registry-secret
    template:
      type: kubernetes.io/dockerconfigjson
  data:
    - secretKey: .dockerconfigjson
      remoteRef:
        key: secret/gitlab/registry
        property: dockerconfigjson

4. GitLab CI — always tag with commit SHA, never `latest`

# .gitlab-ci.yml
build-image:
  stage: build
  script:
    - docker build -t registry.gitlab.com/myorg/myapp/nginx:${CI_COMMIT_SHORT_SHA} .
    - docker push registry.gitlab.com/myorg/myapp/nginx:${CI_COMMIT_SHORT_SHA}
    # Optionally also tag as a semver release on tags
    - |
      if [ -n "$CI_COMMIT_TAG" ]; then
        docker tag registry.gitlab.com/myorg/myapp/nginx:${CI_COMMIT_SHORT_SHA} \
                   registry.gitlab.com/myorg/myapp/nginx:${CI_COMMIT_TAG}
        docker push registry.gitlab.com/myorg/myapp/nginx:${CI_COMMIT_TAG}
      fi

This guarantees every image in your registry is traceable to an exact commit. latest never exists. The manifest not found error becomes structurally impossible for correctly-built images.