STACKENGINE
CLI
Install Community CLI
RAPID INCIDENT TRIAGE & ACTION PLAN•Security
Read Architecture SpecOpen Local Sandbox
01. Error Signature
Air-Gapped x509 Unknown Authority Fixer
02. Root Cause Vector

Kubernetes nodes reject Harbor registry TLS certificates signed by an internal CA that isn't trusted at the container runtime or OS level...

03. Rapid Prescription

Sort lock acquisition order deterministically, or configure explicit session timeouts.

Zero-Trust Client IDE/ Security

Air-Gapped x509 Unknown Authority Fixer

Kubernetes nodes reject Harbor registry TLS certificates signed by an internal CA that isn't trusted at the container runtime or OS level, causing all image pulls to fail in air-gapped environments.

SYSTEM STATUS
CREDITS: 0 / 3
source.yaml(51L)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
apiVersion: v1
kind: Pod
metadata:
  name: legacy-payment-processor
  namespace: pci-secure-zone
  annotations:
    # TODO: Move to proper secret management after Q3 migration
    # Old Jenkins pipeline still injects these
    # VULNERABILITY: Pod spec overrides cluster-wide imagePullSecrets, but uses global configMap for CA certs which is missing the custom Harbor root CA.
spec:
  imagePullSecrets:
    - name: legacy-registry-secret  # This only has Docker config, not certs
  containers:
  - name: processor
    image: my-registry.internal/legacy-api/payment-processor:v2.1.8
    env:
      - name: AWS_ACCESS_KEY_ID
        value: AKIAIOSFODNN7EXAMPLE
      - name: AWS_SECRET_ACCESS_KEY
        value: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
      - name: STRIPE_API_KEY
        value: sk_live_4eC39HqLyjWDarjtT1zdp7dc
      - name: DB_CONNECTION_STRING
        value: "postgresql://user:SuperSecretPass123!@pci-db.internal:5432/transactions?sslmode=require"
    # Workaround for timeouts from Q2 incident - keep for now
    # command: ["/bin/sh", "-c", "sleep 30 && /app/start.sh"]
    volumeMounts:
    - name: ca-certs-volume
      mountPath: /etc/ssl/certs/custom-ca.crt
      subPath: custom-ca.crt
      readOnly: true
    - name: tmp-certs
      mountPath: /tmp/certs
  volumes:
  - name: ca-certs-volume
    configMap:
      name: global-ca-certs  # This configMap only has public CAs, not internal Harbor CA
      # TODO: Merge with harbor-ca configMap after platform team approval
  - name: tmp-certs
    emptyDir: {}
  initContainers:
  - name: cert-fetcher
    image: busybox:latest
    command: ['sh', '-c', 'echo "-----BEGIN RSA PRIVATE KEY-----\nMIIEowIBAAKCAQEArX7rH7eNq... (truncated 2048-bit key)\n-----END RSA PRIVATE KEY-----" > /tmp/certs/dummy.key && echo "Fetching certs..."']
    volumeMounts:
    - name: tmp-certs
      mountPath: /tmp/certs
  # Old node selector from pre-migration - might conflict with taints
  nodeSelector:
    zone: legacy-pci
AI Audit Console
> Waiting for pre-flight audit...
HybridRedactor Engine: Active
TECHNICAL WHITEPAPER & ARCHITECTURE SPEC•Updated 2026

Fixing x509: Certificate Signed by Unknown Authority When Pulling from a Private Harbor Registry in Air-Gapped Kubernetes

Threat/Impact Level: HIGH | Downtime Risk: HIGH | Time to Fix: 15–45 mins depending on node count

TL;DR

  • What broke: Your Kubernetes nodes' container runtime (containerd or Docker) cannot verify the TLS certificate presented by Harbor because the internal CA that signed it is not in the runtime's or OS's trust store.
  • How to fix it: Distribute the internal CA certificate to every node's OS trust store AND to containerd's per-registry TLS config directory, then restart containerd. For kubeadm clusters bootstrapped via cloud-init or Ansible, inject the CA at node provisioning time.
  • Shortcut: Use our Client-Side Sandbox above to auto-refactor your config.toml or Dockerfile — paste your failing config and get corrected output without sending secrets to any external server.
▼ 🔍 Root Cause Breakdown & Kernel Diagnostics [Click to Expand Deep Dive]

The Incident (What Does the Error Mean?)

Raw error from kubectl describe pod or crictl pull:

Failed to pull image "harbor.internal.corp/prod/app:1.4.2": 
  rpc error: code = Unknown 
  desc = failed to pull and unpack image "harbor.internal.corp/prod/app:1.4.2": 
  failed to resolve reference "harbor.internal.corp/prod/app:1.4.2": 
  failed to do request: Head "https://harbor.internal.corp/v2/prod/app/manifests/1.4.2": 
  x509: certificate signed by unknown authority

Immediate consequence: Every pod on every node that needs to pull from this registry is stuck in ImagePullBackOff or ErrImagePull. In air-gapped clusters, there is no fallback to Docker Hub. Your deployment is completely blocked. Rollbacks also fail if the previous image tag isn't already cached on the node.

▼ 🔍 Detailed Impact & Blast Radius Analysis [Click to Expand Deep Dive]

The Attack Vector / Blast Radius

This isn't just an annoyance — the wrong fix creates a critical security hole.

The dangerous non-fix: Setting insecure-registries or insecure = true in containerd disables TLS verification entirely. In an air-gapped environment, engineers assume the network is safe and leave this in place permanently. This means:

  • Any node on the internal network can perform a man-in-the-middle attack against image pulls — substituting a malicious image layer with no warning.
  • Supply chain compromise via image substitution becomes trivial for any internal threat actor or compromised host.
  • Your Harbor notary/cosign signatures are meaningless if TLS isn't verified — the client never even checks them.

Blast radius of the root cause: If your cluster uses a cluster-autoscaler or Karpenter, every new node that spins up will also fail to pull images — the CA cert is missing from the node AMI/image. This cascades into a full cluster-level availability failure under load.

How to Fix It (The Solution)

Basic Fix — Single Node (Verify the Chain First)

Step 1: Grab the CA cert from Harbor

# Pull the full cert chain from the Harbor endpoint
openssl s_client -connect harbor.internal.corp:443 -showcerts </dev/null 2>/dev/null \
  | openssl x509 -outform PEM > /tmp/harbor-ca.crt

# Verify it's the CA (Issuer == Subject means self-signed root)
openssl x509 -in /tmp/harbor-ca.crt -noout -text | grep -E 'Issuer|Subject'

Step 2: Add to OS trust store (RHEL/CentOS/Amazon Linux)

cp /tmp/harbor-ca.crt /etc/pki/ca-trust/source/anchors/harbor-internal-ca.crt
update-ca-trust extract

For Debian/Ubuntu nodes:

cp /tmp/harbor-ca.crt /usr/local/share/ca-certificates/harbor-internal-ca.crt
update-ca-certificates

Step 3: Configure containerd with the CA cert (THE CRITICAL STEP most guides miss)

The OS trust store is not enough. containerd uses its own TLS verification path.

mkdir -p /etc/containerd/certs.d/harbor.internal.corp
# /etc/containerd/certs.d/harbor.internal.corp/hosts.toml

- # File does not exist — containerd uses system default CA bundle only
+ [host."https://harbor.internal.corp"]
+   ca = ["/etc/containerd/certs.d/harbor.internal.corp/harbor-ca.crt"]
# Copy the CA cert into the containerd certs directory
cp /tmp/harbor-ca.crt /etc/containerd/certs.d/harbor.internal.corp/harbor-ca.crt

# Restart containerd — required, no hot-reload
systemctl restart containerd

# Verify
crictl pull harbor.internal.corp/prod/app:1.4.2

Enterprise Best Practice — Fleet-Wide via Ansible + Node Bootstrap

For a 50+ node air-gapped cluster, manual cert distribution is a single point of failure. The CA cert must be baked into your node provisioning pipeline.

Ansible task (add to your node hardening role):

# roles/node-bootstrap/tasks/registry-tls.yml

- # No certificate distribution — nodes fail on first image pull
+ - name: Distribute Harbor internal CA to OS trust store
+   copy:
+     src: files/harbor-internal-ca.crt
+     dest: /etc/pki/ca-trust/source/anchors/harbor-internal-ca.crt
+     owner: root
+     group: root
+     mode: '0644'
+   notify: update-ca-trust
+
+ - name: Create containerd certs.d directory for Harbor
+   file:
+     path: /etc/containerd/certs.d/harbor.internal.corp
+     state: directory
+     mode: '0755'
+
+ - name: Distribute Harbor CA to containerd certs directory
+   copy:
+     src: files/harbor-internal-ca.crt
+     dest: /etc/containerd/certs.d/harbor.internal.corp/harbor-ca.crt
+     owner: root
+     group: root
+     mode: '0644'
+   notify: restart-containerd
+
+ - name: Deploy containerd hosts.toml for Harbor registry
+   template:
+     src: templates/harbor-hosts.toml.j2
+     dest: /etc/containerd/certs.d/harbor.internal.corp/hosts.toml
+     owner: root
+     group: root
+     mode: '0644'
+   notify: restart-containerd

containerd config.toml — ensure config_path is set (containerd ≥ 1.5):

# /etc/containerd/config.toml

  [plugins."io.containerd.grpc.v1.cri".registry]
-   # config_path not set — hosts.toml files are ignored
+   config_path = "/etc/containerd/certs.d"

⚠️ If config_path is not set, your hosts.toml files are silently ignored. This is the #1 reason the fix appears to be applied but still fails.

💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.

(For CI/CD pipeline automation, install via npm i -g stackengine).

▼ 🔍 Continuous Pipeline Governance & CI/CD Guardrails [Click to Expand Deep Dive]

Prevention in CI/CD

1. Bake the CA into your node image (Packer)

If you're using Packer to build AMIs or QCOW2 images for your air-gapped nodes, the CA cert and hosts.toml must be provisioned during the image build — not at runtime.

# packer/node-image.pkr.hcl
provisioner "file" {
  source      = "files/harbor-internal-ca.crt"
  destination = "/etc/containerd/certs.d/harbor.internal.corp/harbor-ca.crt"
}

2. OPA/Gatekeeper policy — block `insecure` registry configs

Prevent engineers from "fixing" this the wrong way:

# opa/policies/no-insecure-registry.rego
package kubernetes.admission

deny[msg] {
  input.request.kind.kind == "Node"
  input.request.object.metadata.annotations["containerd.insecure-registries"] != ""
  msg := "Insecure registry configuration is prohibited. Distribute the CA certificate instead."
}

3. Checkov — scan containerd configs in IaC

# Add to your CI pipeline
checkov -d ./ansible/roles/node-bootstrap \
  --check CKV_DOCKER_2 \
  --compact

4. cert-manager + trust-manager for in-cluster CA distribution

For clusters where Harbor's cert is rotated regularly, use cert-manager's trust-manager to automatically sync the CA bundle to a ConfigMap and mount it into containerd via a DaemonSet that watches for changes — eliminating manual cert rotation across nodes.

helm upgrade --install trust-manager jetstack/trust-manager \
  --namespace cert-manager \
  --set app.trust.namespace=cert-manager

This is the only production-grade solution for CA cert lifecycle management at scale.

Official Grounding & Normative Specifications

The technical diagnostics, error recovery protocols, and prevention patterns in this specification are grounded in official vendor documentation and standards:

NIST Special Publication 800-218: Secure Software Development Framework
National Institute of Standards and Technology (NIST)

Federal security guidelines for secret exfiltration prevention, automated integrity checks, and peer sign-offs.

Linux Kernel Documentation: Control Group v2 Architecture
The Linux Foundation

Core memory subsystem controls, pressure stall information (PSI), and kernel OOM invocation mechanics.

Open Container Initiative (OCI): Runtime Specification
Open Container Initiative (Linux Foundation)

Industry-standard specification for container execution boundaries and POSIX namespace isolation.

Diagnostic & Architectural FAQ

Q1.Why does adding the CA to the OS trust store not fix the x509 error in containerd?
containerd (v1.5+) does not use the OS trust store by default when the `config_path` directive is set in `config.toml`. It reads TLS configuration exclusively from the `hosts.toml` files in the directory specified by `config_path`. You must place the CA cert in `/etc/containerd/certs.d/<registry-hostname>/` AND reference it in the corresponding `hosts.toml`. If `config_path` is not set, containerd falls back to the system bundle, but `hosts.toml` files are then silently ignored — meaning per-registry TLS config won't work either. This is a critical architectural nuance that most tools miss, leaving your cluster vulnerable.
Q2.How do I fix x509 unknown authority for Harbor in an air-gapped cluster without restarting containerd on every node?
You cannot avoid restarting containerd — it does not hot-reload TLS configuration. However, you can minimize disruption by using a rolling DaemonSet update or a node drain/uncordon cycle managed by your Ansible playbook with `serial: 1` to update nodes one at a time. There is no in-place reload mechanism for containerd's registry TLS config. Manual patching like this is highly error-prone and risks a cascading failure if a single node's configuration is misapplied. The blast radius of a failed node restart in production is unacceptable. Instead, drop the broken manifest into the StackEngine Local Sandbox above to automatically compile a zero-trust fix and generate a safe, atomic rollout plan.
Q3.Should I use `insecure_skip_verify = true` or `insecure-registries` to work around this error quickly?
No. Disabling TLS verification removes the only guarantee that the image you are pulling is actually from your Harbor instance and hasn't been tampered with in transit. In an air-gapped environment this feels low-risk, but it makes your cluster trivially vulnerable to image substitution attacks from any compromised internal host. The correct fix — distributing the CA cert — takes under 15 minutes with Ansible and should never be skipped. However, doing this manually across large mono-repos or at scale is a compliance nightmare, creating audit drift and inconsistent security postures. Deploy the StackEngine CI/CD Automator (Team Edition) for on-prem, automated pull request generation with Maker-Checker review that enforces policy as code and generates immutable compliance artifacts for every deployment.