STACKENGINE
CLI
Install Community CLI
RAPID INCIDENT TRIAGE & ACTION PLAN•Security
Read Architecture SpecOpen Local Sandbox
01. Error Signature
Kubelet Insecure TLS Metrics Server Fix
02. Root Cause Vector

Metrics Server fails silently on kubectl top pods because it cannot verify kubelet TLS certs, requiring the --kubelet-insecure-tls flag t...

03. Rapid Prescription

Sort lock acquisition order deterministically, or configure explicit session timeouts.

Zero-Trust Client IDE/ Security

Kubelet Insecure TLS Metrics Server Fix

Metrics Server fails silently on kubectl top pods because it cannot verify kubelet TLS certs, requiring the --kubelet-insecure-tls flag to bypass certificate validation.

SYSTEM STATUS
CREDITS: 0 / 3
source.yaml(75L)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
# TODO: Migrate to new metrics-server once PCI zone allows external traffic
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  annotations:
    "compliance.pci-secure-zone": "legacy-approval-2022-01"
spec:
  replicas: 2
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      containers:
      - name: metrics-server
        image: my-registry.internal/legacy-api/metrics-server:v0.6.2
        args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --metric-resolution=30s
        # VULNERABILITY: Missing --kubelet-insecure-tls flag, causing TLS verification failure with our self-signed kubelet certs
        # TODO: Fix this workaround after the cert rotation next quarter
        env:
        - name: AWS_ACCESS_KEY_ID
          value: "AKIAIOSFODNN7EXAMPLE"
        - name: AWS_SECRET_ACCESS_KEY
          value: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
        - name: STRIPE_API_KEY
          value: "sk_live_4eC39HqLyjWDarjtT1zdp7dc"
        - name: DB_URI
          value: "postgresql://app_user:SuperSecretPass123!@pci-db.internal:5432/legacy_metrics?sslmode=require"
        ports:
        - name: main-port
          containerPort: 4443
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /livez
            port: main-port
            scheme: HTTPS
        readinessProbe:
          httpGet:
            path: /readyz
            port: main-port
            scheme: HTTPS
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp
      volumes:
      - name: tmp-dir
        emptyDir: {}
      # Old config from v0.4.3 - kept for rollback reference
      # - name: certs
      #   secret:
      #     secretName: metrics-server-certs
      priorityClassName: system-cluster-critical
      tolerations:
      - key: "CriticalAddonsOnly"
        operator: "Exists"
      nodeSelector:
        node-role.kubernetes.io/master: ""
      # TODO: Remove after migrating to node affinity
      # securityContext:
      #   runAsUser: 1000
      #   runAsGroup: 3000
      #   fsGroup: 2000
AI Audit Console
> Waiting for pre-flight audit...
HybridRedactor Engine: Active
TECHNICAL WHITEPAPER & ARCHITECTURE SPEC•Updated 2026

How to Fix kubectl top pods Showing Empty Metrics: Metrics Server kubelet-insecure-tls Configuration

Threat/Impact Level: MEDIUM | Exploitability/Downtime Risk: LOW (operational blind spot, not direct RCE) | Time to Fix: 5 mins

TL;DR

  • What broke: Metrics Server pods are running but kubectl top pods/nodes returns error: metrics not available yet or hangs empty — because Metrics Server cannot scrape kubelet /metrics/resource endpoints over TLS without valid cert verification.
  • How to fix it: Add --kubelet-insecure-tls (dev/non-prod) or mount proper CA certs and use --kubelet-certificate-authority (production) to the Metrics Server Deployment args.
  • Shortcut: Use our Client-Side Sandbox above to auto-refactor this — paste your Metrics Server manifest and get the corrected spec without sending your cluster config to any external server.
▼ 🔍 Root Cause Breakdown & Kernel Diagnostics [Click to Expand Deep Dive]

The Incident (What does the error mean?)

You run kubectl top pods -n production and get:

Error from server (ServiceUnavailable): the server is currently unable to handle the request (get pods.metrics.k8s.io)

or the HPA silently stops scaling because it has no CPU/memory signal. Metrics Server logs tell the real story:

E0612 14:32:01.783204       1 scraper.go:140] "Failed to scrape node" err="Get \"https://192.168.1.10:10250/metrics/resource\": x509: certificate signed by unknown authority" node="worker-node-01"

Immediate consequence: HPA controllers go blind. kubectl top is dead. Any autoscaling tied to CPU/memory metrics stops functioning. In clusters using kubeadm or self-managed CA, kubelet serving certs are signed by the cluster CA — which Metrics Server doesn't trust by default because it runs with its own TLS context and no CA bundle is injected.

▼ 🔍 Detailed Impact & Blast Radius Analysis [Click to Expand Deep Dive]

The Attack Vector / Blast Radius

This is an operational security misconfiguration, not a direct exploit vector — but the blast radius is significant:

  1. Autoscaler blindness: HPA cannot retrieve metrics.k8s.io API objects. Pods under load will not scale out. You get cascading OOM kills or latency spikes with zero automated remediation.
  2. The insecure bypass risk: The common "just add --kubelet-insecure-tls" fix disables TLS verification entirely between Metrics Server and every kubelet endpoint. In a multi-tenant cluster or one exposed to a compromised internal network, a MITM attacker on the pod network could serve fake metrics — causing HPAs to scale down healthy workloads or refuse to scale up under real load. This is a metrics poisoning attack surface.
  3. Compliance failure: PCI-DSS, SOC2, and CIS Kubernetes Benchmark (Control 4.2.10) explicitly require kubelet serving certificate verification. Running --kubelet-insecure-tls in production is an automatic audit finding.

How to Fix It (The Solution)

Basic Fix (Dev / Non-Prod Only)

Add --kubelet-insecure-tls to the Metrics Server container args. This skips x509 verification against kubelet certs.

# metrics-server Deployment spec.containers[0].args
  args:
    - --cert-dir=/tmp
    - --secure-port=4443
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
+   - --kubelet-insecure-tls

If deploying via Helm:

# values.yaml
 defaultArgs:
-  - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
+  - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
+  - --kubelet-insecure-tls

Do not use this in production. Full stop.

Enterprise Best Practice (Production)

The correct fix is to provision valid kubelet serving certificates signed by a CA that Metrics Server trusts, then mount that CA into the Metrics Server pod.

Step 1: Ensure kubelets are using properly signed serving certs. With kubeadm, enable server cert rotation:

# kubelet-config.yaml (on each node)
  serverTLSBootstrap: true
+ rotateCertificates: true

Approve pending CSRs:

kubectl get csr | grep Pending | awk '{print $1}' | xargs kubectl certificate approve

Step 2: Mount the cluster CA into Metrics Server and reference it:

# metrics-server Deployment
  spec:
    containers:
    - name: metrics-server
      args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP
-       - --kubelet-insecure-tls
+       - --kubelet-certificate-authority=/etc/kubernetes/pki/ca.crt
+     volumeMounts:
+       - name: cluster-ca
+         mountPath: /etc/kubernetes/pki/ca.crt
+         readOnly: true
+   volumes:
+     - name: cluster-ca
+       hostPath:
+         path: /etc/kubernetes/pki/ca.crt
+         type: File

Alternatively, use a projected ConfigMap containing the CA bundle instead of a hostPath mount — hostPath is itself a security finding in hardened clusters.

💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.

(For CI/CD pipeline automation, install via npm i -g stackengine).

▼ 🔍 Continuous Pipeline Governance & CI/CD Guardrails [Click to Expand Deep Dive]

Prevention in CI/CD

1. Conftest / OPA policy — block insecure-tls in production namespaces:

# policy/deny_insecure_metrics_server.rego
package main

deny[msg] {
  input.kind == "Deployment"
  input.metadata.namespace == "kube-system"
  container := input.spec.template.spec.containers[_]
  contains(container.name, "metrics-server")
  arg := container.args[_]
  arg == "--kubelet-insecure-tls"
  msg := "POLICY VIOLATION: metrics-server --kubelet-insecure-tls is forbidden in production. Use --kubelet-certificate-authority instead."
}

2. Checkov inline scan in your pipeline:

checkov -f metrics-server-deployment.yaml --check CKV_K8S_28,CKV_K8S_30

3. Kube-bench scheduled job: Run kube-bench node as a CronJob targeting control 4.2.10 to continuously assert kubelet serving cert configuration hasn't drifted.

4. Helm chart pinning: Lock your metrics-server chart version in your GitOps repo (fleet.yaml or helmrelease.yaml) and gate upgrades through a PR policy that requires the OPA policy check to pass in CI before merge.

Official Grounding & Normative Specifications

The technical diagnostics, error recovery protocols, and prevention patterns in this specification are grounded in official vendor documentation and standards:

NIST Special Publication 800-218: Secure Software Development Framework
National Institute of Standards and Technology (NIST)

Federal security guidelines for secret exfiltration prevention, automated integrity checks, and peer sign-offs.

Linux Kernel Documentation: Control Group v2 Architecture
The Linux Foundation

Core memory subsystem controls, pressure stall information (PSI), and kernel OOM invocation mechanics.

Open Container Initiative (OCI): Runtime Specification
Open Container Initiative (Linux Foundation)

Industry-standard specification for container execution boundaries and POSIX namespace isolation.

Diagnostic & Architectural FAQ

Q1.Why does kubectl top pods show 'error: metrics not available yet' even though Metrics Server pods are Running?
The root cause is a broken TLS trust chain. A Running pod only indicates the container started, not that it can successfully scrape kubelets. Metrics Server fails with an x509 error when connecting to kubelet port 10250 because it doesn't trust the kubelet's self-signed certificate. This is a fundamental misconfiguration in the cluster's observability layer. To build authority, you must diagnose this precisely: check the Metrics Server logs for 'certificate signed by unknown authority'. The manual fix involves modifying the Metrics Server deployment arguments, which is where most teams introduce risk.
Q2.Is --kubelet-insecure-tls safe to use in a production Kubernetes cluster?
Absolutely not. Using --kubelet-insecure-tls is a catastrophic security bypass that disables all TLS verification between Metrics Server and every kubelet. This exposes your cluster to metrics poisoning attacks—an attacker on your internal network can inject fabricated CPU/memory data, causing Horizontal Pod Autoscalers to make insane scaling decisions and potentially crash your applications. It violates CIS Benchmark 4.2.10 and will fail any serious audit (SOC2, PCI-DSS). Manual patching of manifests is highly error-prone and the blast radius is your entire cluster. Do not edit YAML by hand. Instead, drop the broken manifest into the StackEngine Local Sandbox above to automatically compile a zero-trust fix that uses proper certificate authority validation.
Q3.How do I get Metrics Server working on a kubeadm cluster without disabling TLS verification?
The enterprise-grade method requires enabling kubelet server certificate rotation (serverTLSBootstrap: true), restarting kubelets, approving CSRs, and correctly mounting the cluster CA into the Metrics Server pod with the --kubelet-certificate-authority argument. While technically possible, performing this manually across multiple clusters or environments is a compliance nightmare—one missed CSR or mis-mounted volume breaks metrics globally. At scale, this manual process is unsustainable and audit-proofing it is impossible. For reliable, automated PR drafting under Maker-Checker review across large mono-repos or fleets, upgrade to the StackEngine CI/CD Automator (Team Edition). It provides on-prem, policy-driven fixes integrated directly into your CI/CD pipeline, ensuring every deployment is automatically secured and compliant under Maker-Checker review.