Fixing Helm 'manifest for nginx:latest not found' Errors with Private GitLab Registry Image Pull Secrets

Threat/Impact Level: HIGH | Downtime Risk: HIGH | Time to Fix: 15–30 mins

---

TL;DR

• What broke: Kubernetes cannot pull the image because either the tag (latest or a custom tag) does not exist in the private GitLab registry, the imagePullSecret is missing/malformed, or the Helm chart is referencing the wrong registry path.
• How to fix it: Verify the exact image digest exists in GitLab, create a valid kubernetes.io/dockerconfigjson secret, and wire it into the Helm chart's imagePullSecrets block with a pinned, immutable tag.
• Shortcut: Use our Client-Side Sandbox below to auto-refactor your failing values.yaml and regenerate the pull secret manifest without leaking your registry token.

---

The Incident (What Does the Error Mean?)

This is what kubectl describe pod <pod-name> shows during the outage:

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  2m                 default-scheduler  Successfully assigned default/app-6d7f9b-xkq2p to node-1
  Normal   Pulling    2m                 kubelet            Pulling image "registry.gitlab.com/myorg/myapp/nginx:latest"
  Warning  Failed     112s               kubelet            Failed to pull image "registry.gitlab.com/myorg/myapp/nginx:latest": rpc error: code = NotFound desc = failed to pull and unpack image "registry.gitlab.com/myorg/myapp/nginx:latest": failed to resolve reference "registry.gitlab.com/myorg/myapp/nginx:latest": unexpected status code 401 Unauthorized
  Warning  Failed     112s               kubelet            Error: ErrImagePull
  Warning  BackOff    98s (x3 over 110s) kubelet            Back-off pulling image "registry.gitlab.com/myorg/myapp/nginx:latest"
  Warning  Failed     98s (x3 over 110s) kubelet            Error: ImagePullBackOff

Immediate consequence: The pod never reaches Running state. It oscillates between ErrImagePull and ImagePullBackOff. Every replica in the Deployment is dead. Your service is down.

There are three distinct failure modes that produce this identical error surface:

1. Tag does not exist — nginx:latest was never pushed to registry.gitlab.com/myorg/myapp/, or was deleted/overwritten and the manifest digest is gone.
2. imagePullSecret is absent or bound to the wrong namespace — The kubelet cannot authenticate to the private registry at all (401/403).
3. Helm chart image values are wrong — image.repository or image.tag in values.yaml points to a non-existent path or uses a stale override.

---

The Attack Vector / Blast Radius

This is not just an ops inconvenience. The blast radius is significant:

Deployment blast radius:

• All pods in the Deployment/StatefulSet/DaemonSet referencing this image are unschedulable. A rolling update leaves you with zero healthy replicas if the old ReplicaSet was already scaled down.
• Horizontal Pod Autoscaler scale-out events will also fail silently — new pods spin up into ImagePullBackOff under load.

Security angle — why latest is a liability:

• Using latest means your CI/CD pipeline can silently push a broken or malicious layer to the tag and every future pod restart pulls it automatically. There is no pinning, no auditability, no rollback path via image tag alone.
• In a supply chain attack scenario (e.g., a compromised GitLab CI runner), an attacker can overwrite latest with a backdoored image. Because imagePullPolicy: Always re-pulls on every pod restart, the malicious image propagates to all nodes on the next rollout or node failure recovery — without any Helm release change being made.
• A missing or overly-permissive imagePullSecret stored as a plain Kubernetes Secret with no RBAC restriction means any pod in the namespace can mount it and exfiltrate your GitLab registry credentials.

---

How to Fix It (The Solution)

Step 1 — Verify the image tag actually exists

# Authenticate and check the tag exists in GitLab registry
curl -s --header "PRIVATE-TOKEN: <your-pat>" \
  "https://registry.gitlab.com/v2/myorg/myapp/nginx/tags/list"

# Or via the GitLab API
curl -s --header "PRIVATE-TOKEN: <your-pat>" \
  "https://gitlab.com/api/v4/projects/<project-id>/registry/repositories"

If latest is not in the tag list, your CI pipeline never pushed it, or it was garbage-collected. Fix the pipeline first.

---

Step 2 — Create the imagePullSecret correctly

# Create the secret in the SAME namespace as your Helm release
kubectl create secret docker-registry gitlab-registry-secret \
  --docker-server=registry.gitlab.com \
  --docker-username=<gitlab-deploy-token-username> \
  --docker-password=<gitlab-deploy-token-or-pat> \
  [email protected] \
  --namespace=<your-release-namespace>

⚠️ Use a GitLab Deploy Token (scoped to read_registry only), NOT your personal access token. Rotate it. Store it in Vault or a sealed secret — not in plaintext in your repo.

---

Step 3 — Fix the Helm values.yaml (Basic Fix vs. Enterprise Best Practice)

Bad configuration (what's breaking you right now):

- image:
-   repository: nginx
-   tag: latest
-   pullPolicy: Always
- imagePullSecrets: []

Corrected configuration (Enterprise Best Practice):

+ image:
+   repository: registry.gitlab.com/myorg/myapp/nginx
+   tag: "1.25.3"   # NEVER use 'latest' in production. Pin to immutable tag or SHA digest.
+   pullPolicy: IfNotPresent  # Avoid Always unless you have a hard requirement
+ imagePullSecrets:
+   - name: gitlab-registry-secret

Even better — pin to the image SHA digest in your Helm values (immutable, tamper-evident):

+ image:
+   repository: registry.gitlab.com/myorg/myapp/nginx
-   tag: "1.25.3"
+   tag: ""  # leave empty when using digest
+   digest: "sha256:a3b4c1d2e5f6..."  # output of: docker inspect --format='{{index .RepoDigests 0}}' <image>
+   pullPolicy: IfNotPresent
+ imagePullSecrets:
+   - name: gitlab-registry-secret

Note: Digest pinning requires your Helm chart's deployment template to support image.digest. If it doesn't, patch the template to render repository@digest when digest is set.

---

Step 4 — Verify the secret is in the right namespace and re-deploy

# Confirm secret exists in correct namespace
kubectl get secret gitlab-registry-secret -n <your-namespace> -o jsonpath='{.type}'
# Expected output: kubernetes.io/dockerconfigjson

# Validate the encoded credentials are correct
kubectl get secret gitlab-registry-secret -n <your-namespace> \
  -o jsonpath='{.data.\.dockerconfigjson}' | base64 --decode | jq .

# Re-deploy the Helm release
helm upgrade --install my-release ./my-chart \
  --namespace <your-namespace> \
  --set image.tag="1.25.3" \
  --set imagePullSecrets[0].name=gitlab-registry-secret

# Watch pods come up
kubectl rollout status deployment/my-release -n <your-namespace>

---

💡 Tired of pasting proprietary configs into ChatGPT? Generic AI tools log your company's ARNs, DB strings, and private keys. StackEngine is a zero-backend, pure Client-Side WASM utility. Drop your failing config into the sandbox above. We redact your secrets locally in the browser and auto-generate the refactored code using your own API key.

---

Prevention in CI/CD

Stop this from ever hitting production again. These controls are non-negotiable for any team past 5 engineers.

1. Enforce non-latest tags with OPA/Gatekeeper

Deploy this ConstraintTemplate to block any image referencing latest:

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8sdenylatesttag
spec:
  crd:
    spec:
      names:
        kind: K8sDenyLatestTag
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8sdenylatesttag
        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          endswith(container.image, ":latest")
          msg := sprintf("Container '%v' uses ':latest' tag. Pin to an immutable tag or digest.", [container.name])
        }
        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          not contains(container.image, ":")
          msg := sprintf("Container '%v' has no image tag. Explicit tag or digest required.", [container.name])
        }

2. Checkov scan in your GitLab CI pipeline

# .gitlab-ci.yml
checkov-scan:
  stage: validate
  image: bridgecrew/checkov:latest
  script:
    - checkov -d ./helm-chart/templates --check CKV_K8S_14,CKV_K8S_15
    # CKV_K8S_14: Image Tag should be fixed - not latest
    # CKV_K8S_15: Image should use Always pull policy (paired with digest pinning)
  allow_failure: false

3. Automate secret rotation with External Secrets Operator

Never manually kubectl create secret again. Use External Secrets Operator to sync your GitLab deploy token from HashiCorp Vault or AWS Secrets Manager directly into the namespace as a kubernetes.io/dockerconfigjson secret — with automatic rotation.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: gitlab-registry-secret
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: gitlab-registry-secret
    template:
      type: kubernetes.io/dockerconfigjson
  data:
    - secretKey: .dockerconfigjson
      remoteRef:
        key: secret/gitlab/registry
        property: dockerconfigjson

4. GitLab CI — always tag with commit SHA, never latest

# .gitlab-ci.yml
build-image:
  stage: build
  script:
    - docker build -t registry.gitlab.com/myorg/myapp/nginx:${CI_COMMIT_SHORT_SHA} .
    - docker push registry.gitlab.com/myorg/myapp/nginx:${CI_COMMIT_SHORT_SHA}
    # Optionally also tag as a semver release on tags
    - |
      if [ -n "$CI_COMMIT_TAG" ]; then
        docker tag registry.gitlab.com/myorg/myapp/nginx:${CI_COMMIT_SHORT_SHA} \
                   registry.gitlab.com/myorg/myapp/nginx:${CI_COMMIT_TAG}
        docker push registry.gitlab.com/myorg/myapp/nginx:${CI_COMMIT_TAG}
      fi

This guarantees every image in your registry is traceable to an exact commit. latest never exists. The manifest not found error becomes structurally impossible for correctly-built images.