Fixing SealedSecrets 'controller failed to unseal': Private Key Mismatch Diagnosis and Recovery

Threat/Impact Level: CRITICAL | Exploitability/Downtime Risk: HIGH | Time to Fix: 15–45 mins depending on key backup availability

TL;DR

• What broke: The SealedSecrets controller's active private key does not match the key that was used to encrypt the SealedSecret resource — decryption fails hard, secret is never mounted, pod stays in CreateContainerConfigError or Pending.
• How to fix it: Identify the correct key fingerprint, restore the original private key secret into kube-system, restart the controller, or re-seal all affected secrets against the current cluster public key.
• Shortcut: Use our Client-Side Sandbox above to paste your SealedSecret YAML and controller logs — it will auto-identify the fingerprint mismatch and generate the corrected kubeseal re-encryption commands locally in your browser.

---

The Incident (What Does the Error Mean?)

Raw controller log output:

E0612 03:14:22.901847       1 controller.go:166] failed to unseal secret default/db-credentials: no key could decrypt secret
E0612 03:14:22.901901       1 controller.go:166] failed to unseal secret production/api-keys: no key could decrypt secret

Or the variant with explicit fingerprint rejection:

error unsealing secret: crypto/rsa: decryption error
no private key matched fingerprint: a3:f1:9c:44:...

Immediate consequence: Every SealedSecret object encrypted against the old key becomes permanently unreadable by the current controller. Kubernetes Secret objects are never created. Every workload referencing those secrets — Deployments, StatefulSets, CronJobs — fails to start. In production, this is a full application outage.

---

The Attack Vector / Blast Radius

This failure mode has two distinct causes, both catastrophic:

Cause 1 — Cluster rebuild / DR failover without key backup. The SealedSecrets controller auto-generates a fresh RSA-4096 key pair on first boot. If you rebuilt the cluster (disaster recovery, migration, cluster upgrade gone wrong) without exporting and re-importing the original sealed-secrets-key secret from kube-system, every single previously-sealed secret in your GitOps repo is now permanently unrecoverable without the plaintext originals. Your entire secret store is bricked.

Cause 2 — Accidental key rotation or controller reinstall. Helm upgrade --force, deleting the controller pod's persistent key secret, or reinstalling via a Helm chart that doesn't preserve existing keys generates a new key. Same result.

Blast radius:

• All namespaces using SealedSecrets lose secret injection simultaneously.
• CI/CD pipelines that apply SealedSecret manifests from Git will appear to succeed (kubectl apply returns OK) but the controller silently fails to decrypt — no alerting by default.
• If you store DB credentials, API keys, or TLS certs as SealedSecrets, every downstream service is down.
• If the original private key is gone and you have no plaintext backup, the secrets are cryptographically unrecoverable. You must rotate all credentials from scratch.

---

How to Fix It

Step 0 — Confirm the mismatch

# Get the fingerprint the SealedSecret was encrypted against
kubectl get sealedsecret db-credentials -n default -o jsonpath='{.spec.encryptedData}' | \
  kubeseal --validate --controller-namespace kube-system 2>&1

# List all private keys currently known to the controller
kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key

If the fingerprint in the error does not appear in the controller's key list, the key is missing.

---

Basic Fix — Restore the original private key from backup

If you exported the key before the incident (you should have — see Prevention below):

# Re-import the backed-up key into kube-system
kubectl apply -f sealed-secrets-master-key-backup.yaml -n kube-system

# Restart controller to pick up the restored key
kubectl rollout restart deployment/sealed-secrets-controller -n kube-system

# Verify controller now lists the restored key
kubectl logs -n kube-system deployment/sealed-secrets-controller | grep -i "key"

---

Enterprise Best Practice — Re-seal all secrets against the new cluster key

If the original key is gone, you must re-seal from plaintext. This requires you to have the original secret values (from a vault, HSM, or password manager — not from the cluster).

- # Old workflow: seal once, commit, forget the plaintext
- kubeseal --format yaml < secret-plain.yaml > sealed-secret.yaml
- # Result: sealed against a key that no longer exists

+ # Correct enterprise workflow: always seal against the CURRENT cluster's public cert
+ # Step 1: Export the active public cert explicitly
+ kubeseal --fetch-cert \
+   --controller-namespace kube-system \
+   --controller-name sealed-secrets-controller \
+   > current-cluster-pub-cert.pem
+
+ # Step 2: Re-seal using the fetched cert (works offline, no cluster API dependency)
+ kubeseal --format yaml \
+   --cert current-cluster-pub-cert.pem \
+   < secret-plain.yaml > sealed-secret-new.yaml
+
+ # Step 3: Commit sealed-secret-new.yaml to Git, delete the old one
+ # Step 4: Store current-cluster-pub-cert.pem in your secrets manager for DR

For scope-hardening, always use --scope namespace-wide or --scope strict (default) to prevent cross-namespace secret theft:

- kubeseal --format yaml < secret.yaml > sealed.yaml
+ kubeseal --format yaml --scope strict < secret.yaml > sealed.yaml
+ # 'strict' binds the sealed secret to BOTH the name AND namespace
+ # An attacker who can write to another namespace cannot replay this sealed secret

---

💡 Tired of pasting proprietary configs into ChatGPT? Generic AI tools log your company's ARNs, DB strings, and private keys. StackEngine is a zero-backend, pure Client-Side WASM utility. Drop your failing config into the sandbox above. We redact your secrets locally in the browser and auto-generate the refactored code using your own API key.

---

Prevention in CI/CD

1. Backup the master key on every cluster provisioning run — non-negotiable.

# Export ALL sealed-secrets keys (run this immediately after cluster bootstrap)
kubectl get secret -n kube-system \
  -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o yaml > sealed-secrets-master-key-backup.yaml

# Encrypt and store in your secrets manager (Vault, AWS Secrets Manager, etc.)
vault kv put secret/k8s/sealed-secrets-master-key \
  [email protected]

2. Add a CI validation step that detects key mismatch before merge:

# .github/workflows/validate-sealed-secrets.yaml
- name: Validate SealedSecrets can be unsealed
  run: |
    kubeseal --validate \
      --controller-namespace kube-system \
      < ${{ matrix.sealed_secret_file }}

This step will fail the PR if the SealedSecret was encrypted against a key the current cluster doesn't hold.

3. Use Checkov policy CKV_K8S_35 and add a custom OPA/Conftest rule:

# conftest policy: deny SealedSecrets without strict scope
deny[msg] {
  input.kind == "SealedSecret"
  not input.metadata.annotations["sealedsecrets.bitnami.com/cluster-wide"]
  not input.spec.template.metadata.namespace
  msg := sprintf("SealedSecret '%v' must declare an explicit namespace scope", [input.metadata.name])
}

4. Terraform/Helm: pin the controller version and use a pre-existing key secret:

- # Helm install with no key preservation — regenerates key on every reinstall
- helm upgrade --install sealed-secrets sealed-secrets/sealed-secrets \
-   -n kube-system

+ # Pre-create the key secret from backup BEFORE installing the controller
+ kubectl apply -f sealed-secrets-master-key-backup.yaml
+ helm upgrade --install sealed-secrets sealed-secrets/sealed-secrets \
+   -n kube-system \
+   --set keyrenewperiod=0  # Disable auto-rotation if using externally managed keys

5. Alert on unseal failures. The controller exposes a Prometheus metric sealed_secrets_controller_unseal_errors_total. Set a > 0 alert with severity: page.