STACKENGINE
CLI
Install Community CLI
RAPID INCIDENT TRIAGE & ACTION PLAN•Performance
Read Architecture SpecOpen Local Sandbox
01. Error Signature
GKE Autopilot Resource Scheduling Resolver
02. Root Cause Vector

GKE Autopilot pod stuck in Pending due to insufficient CPU headroom and a not-ready node taint, blocking all workload scheduling until th...

03. Rapid Prescription

Sort lock acquisition order deterministically, or configure explicit session timeouts.

Zero-Trust Client IDE/ Performance

GKE Autopilot Resource Scheduling Resolver

GKE Autopilot pod stuck in Pending due to insufficient CPU headroom and a not-ready node taint, blocking all workload scheduling until the cluster provisions new capacity.

SYSTEM STATUS
CREDITS: 0 / 3
source.yaml(74L)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
# WARNING: This is a legacy deployment for the PCI-secure-zone 'transaction-processor' service.
# TODO: Migrate to new Helm charts by Q4, but this YAML has been patched for Autopilot compatibility (supposedly).
# Deployed via Jenkins pipeline: 'deploy-legacy-pci'.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: transaction-processor-v2
  namespace: pci-secure-zone
  labels:
    app: transaction-processor
    version: "2.1"
    managed-by: "terraform-0.12"
    # TODO: Clean up old labels from migration
    legacy-tier: "gold"
spec:
  replicas: 3
  selector:
    matchLabels:
      app: transaction-processor
  template:
    metadata:
      labels:
        app: transaction-processor
        # Redundant label for old monitoring
        monitoring-group: "high-priority"
    spec:
      # VULNERABILITY: Explicit nodeSelector for a non-existent label 'pci-zone' combined with resource requests exceeding Autopilot's implicit limits triggers scheduling failure.
      nodeSelector:
        pci-zone: "secure"  # This label doesn't exist on Autopilot nodes; leftover from on-prem config.
      containers:
      - name: processor
        image: my-registry.internal/legacy-api/transaction-processor:stable-2.1
        # TODO: Replace with sidecar pattern for secrets (temporary workaround)
        env:
        - name: DB_CONNECTION_STRING
          value: "postgresql://admin:LegacyDbP@ssw0rd2024!@pci-db.internal:5432/transactions?sslmode=require"
        - name: AWS_ACCESS_KEY_ID
          value: "AKIAIOSFODNN7EXAMPLE"
        - name: AWS_SECRET_ACCESS_KEY
          value: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
        # Stripe key for fallback payments (commented out but still in source)
        # - name: STRIPE_API_KEY
        #   value: "sk_live_4eC39HqLyjWDarjtT1zdp7dc"
        resources:
          requests:
            cpu: "2"  # Autopilot default max per container is 1 CPU; this request is too high.
            memory: "4Gi"
          limits:
            cpu: "3"  # Also exceeds limit, but Autopilot ignores explicit limits; the request is the issue.
            memory: "6Gi"
        # Old health check from pre-migration
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
        # TODO: Add readiness probe after fixing the CPU issue
      # Service account with overly broad permissions for legacy compatibility
      serviceAccountName: legacy-pci-processor
      # Tolerations copied from another deployment (irrelevant here)
      tolerations:
      - key: "node.kubernetes.io/not-ready"
        operator: "Exists"
        effect: "NoSchedule"
        # This toleration allows scheduling on not-ready nodes, but combined with high CPU request and missing nodeSelector, it leads to pending state.
      # Priority class to bump scheduling (workaround for previous issues)
      priorityClassName: high-priority
# Note: RSA key for SSH backup access (used by old bootstrap script)
# -----BEGIN RSA PRIVATE KEY-----
# MIIEowIBAAKCAQEArLJ9J7L7XvL8sQKpz7KXbH6pGqT8nMjRwY1tLmNfGvHqWz
# ... (truncated for brevity) ...
# -----END RSA PRIVATE KEY-----
AI Audit Console
> Waiting for pre-flight audit...
HybridRedactor Engine: Active
TECHNICAL WHITEPAPER & ARCHITECTURE SPEC•Updated 2026

Fixing GKE Autopilot Pods Stuck in Pending: Insufficient CPU & Node Not-Ready Taint

Threat/Impact Level: HIGH | Downtime Risk: HIGH | Time to Fix: 5–15 mins

TL;DR

  • What broke: Your pod's CPU requests exceed available allocatable CPU on the sole ready node, and the only other node is tainted node.kubernetes.io/not-ready, meaning the scheduler has zero valid targets.
  • How to fix it: Right-size your CPU requests to match Autopilot's predefined resource tiers, or add a tolerations block if you intentionally need to schedule onto not-ready nodes during warm-up (rare/advanced).
  • Fast path: Use our Client-Side Sandbox above to auto-refactor your pod spec — paste your YAML, get corrected resources blocks instantly without leaking your config to a third party.
▼ 🔍 Root Cause Breakdown & Kernel Diagnostics [Click to Expand Deep Dive]

The Incident (What Does the Error Mean?)

Raw scheduler event:

0/1 nodes are available:
  1 Insufficient cpu,
  1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.

The Kubernetes scheduler evaluated every node in the cluster and found no viable placement target:

  1. Insufficient cpu — The sum of CPU requests across all pods already scheduled on the ready node, plus your new pod's request, exceeds the node's allocatable CPU. In Autopilot, you don't manage node pools directly, but Autopilot still enforces minimum and maximum resource request tiers. If your request is malformed (e.g., requests.cpu: 0 or an absurdly high value like 32), Autopilot either rejects it or cannot find a matching machine shape fast enough.
  2. node.kubernetes.io/not-ready — A second node exists but is mid-provisioning or unhealthy. Autopilot is likely spinning up a new node in response to the pending pod, but it hasn't passed readiness checks yet. Your pod spec has no toleration for this taint, so the scheduler won't place it there even temporarily.

Immediate consequence: The pod stays in Pending indefinitely if Autopilot provisioning stalls (e.g., quota exhaustion, region capacity, or a misconfigured request that no Autopilot tier can satisfy).

▼ 🔍 Detailed Impact & Blast Radius Analysis [Click to Expand Deep Dive]

The Attack Vector / Blast Radius

This is a cascading availability failure, not a security exploit, but the blast radius is severe in production:

  • Single-replica deployments go fully dark. If this is your only pod for a service, it's a complete outage the moment the previous pod terminates.
  • HPA death spiral. If a Horizontal Pod Autoscaler is active, it may attempt to scale out, creating more Pending pods, which triggers more Autopilot node provisioning requests, which exhausts your GCP project's CPUS quota faster — turning a scheduling hiccup into a quota incident.
  • Autopilot provisioning stall. Autopilot will not provision a node for a CPU request it cannot map to a valid Compute Class. A request of cpu: 0 or a non-standard value like 1500m with a memory ratio that violates Autopilot's constraints will leave the pod Pending forever — no new node will ever appear.
  • Readiness probe failures upstream. Any service depending on this pod (via ClusterIP or Ingress) starts returning 502/503 as soon as the endpoint is deregistered.

How to Fix It (The Solution)

Basic Fix — Right-Size CPU Requests for Autopilot

Autopilot enforces a minimum of 250m CPU per container and requires CPU/memory ratios within specific bounds. The most common cause of Insufficient cpu on Autopilot is either an omitted requests block (defaults to 0, which Autopilot mutates but may conflict) or a value that doesn't align with available node shapes.

Check your current pod spec:

kubectl describe pod <pod-name> -n <namespace>
kubectl get events -n <namespace> --sort-by='.lastTimestamp'

Check Autopilot's mutation of your requests:

kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[*].resources}'
# deployment.yaml — container resources block
 spec:
   containers:
   - name: my-app
     image: gcr.io/my-project/my-app:latest
     resources:
-      requests:
-        cpu: "0"
-        memory: "64Mi"
-      limits:
-        cpu: "0"
-        memory: "64Mi"
+      requests:
+        cpu: "250m"
+        memory: "512Mi"
+      limits:
+        cpu: "500m"
+        memory: "512Mi"

Autopilot CPU/Memory ratio rule: For the general-purpose compute class, memory must be between 1 GiB and 6.5 GiB per vCPU. A request of 250m CPU with 512Mi memory is valid. A request of 250m CPU with 64Mi memory may be rejected or mutated by the Autopilot admission webhook.

Enterprise Best Practice — Compute Classes, PodDisruptionBudgets, and Topology Spread

For production workloads, relying on default Autopilot scheduling is insufficient. Implement explicit compute class targeting and spread constraints.

# deployment.yaml — full production-grade spec
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: my-app
 spec:
+  replicas: 2
   selector:
     matchLabels:
       app: my-app
   template:
     metadata:
       labels:
         app: my-app
+        cloud.google.com/compute-class: "general-purpose"
     spec:
+      topologySpreadConstraints:
+      - maxSkew: 1
+        topologyKey: topology.kubernetes.io/zone
+        whenUnsatisfiable: DoNotSchedule
+        labelSelector:
+          matchLabels:
+            app: my-app
       containers:
       - name: my-app
         image: gcr.io/my-project/my-app:latest
         resources:
-          requests:
-            cpu: "100m"
-            memory: "128Mi"
+          requests:
+            cpu: "500m"
+            memory: "1Gi"
+          limits:
+            cpu: "1000m"
+            memory: "1Gi"
# poddisruptionbudget.yaml
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+  name: my-app-pdb
+spec:
+  minAvailable: 1
+  selector:
+    matchLabels:
+      app: my-app

Why replicas: 2 matters here: With a single replica, any node provisioning event = downtime. Two replicas across two zones means Autopilot's not-ready node taint only affects one pod while the other stays live.

💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.

(For CI/CD pipeline automation, install via npm i -g stackengine).

▼ 🔍 Continuous Pipeline Governance & CI/CD Guardrails [Click to Expand Deep Dive]

Prevention in CI/CD

Don't let malformed resource requests reach your cluster. Enforce this at the pipeline level.

1. Conftest / OPA Policy (blocks zero-CPU requests at `kubectl apply` time)

# policy/autopilot_resources.rego
package autopilot.resources

deny[msg] {
  container := input.spec.template.spec.containers[_]
  cpu_request := container.resources.requests.cpu
  units.parse_quantity(cpu_request) < units.parse_quantity("250m")
  msg := sprintf("Container '%v' CPU request %v is below Autopilot minimum of 250m", [container.name, cpu_request])
}

deny[msg] {
  container := input.spec.template.spec.containers[_]
  not container.resources.requests
  msg := sprintf("Container '%v' has no resource requests defined — required for GKE Autopilot", [container.name])
}

2. Checkov Scan in GitHub Actions

# .github/workflows/checkov.yaml
- name: Checkov Kubernetes Scan
  uses: bridgecrewio/checkov-action@master
  with:
    directory: k8s/
    framework: kubernetes
    check: CKV_K8S_10,CKV_K8S_11,CKV_K8S_12,CKV_K8S_13
    # CKV_K8S_10: CPU requests set
    # CKV_K8S_11: CPU limits set
    # CKV_K8S_12: Memory requests set
    # CKV_K8S_13: Memory limits set

3. Kustomize Overlay for Environment-Specific Resource Floors

# overlays/production/resources-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      containers:
      - name: my-app
        resources:
          requests:
            cpu: "500m"
            memory: "1Gi"
          limits:
            cpu: "1000m"
            memory: "1Gi"

4. Monitor Pending Pods with Cloud Monitoring Alert

# MQL alert policy — fires if any pod is Pending > 5 minutes
fetch k8s_pod
| metric 'kubernetes.io/pod/volume/total_bytes'
| filter (resource.labels.namespace_name != 'kube-system')
# Use GKE Workload metrics: kubernetes.io/pod/status/phase with phase=Pending
# Set threshold: > 0 for 5 consecutive minutes → PagerDuty/Slack

Combine this with a GKE Autopilot quota alert on compute.googleapis.com/cpus in your GCP project to catch regional capacity exhaustion before it cascades.

Official Grounding & Normative Specifications

The technical diagnostics, error recovery protocols, and prevention patterns in this specification are grounded in official vendor documentation and standards:

PostgreSQL Core Docs: Lock Management & Deadlock Detection
PostgreSQL Global Development Group

Official runtime configuration documentation governing lock acquisition timeouts, deadlock detection cycles, and wait logging.

NIST Special Publication 800-218: Secure Software Development Framework
National Institute of Standards and Technology (NIST)

Federal security guidelines for secret exfiltration prevention, automated integrity checks, and peer sign-offs.

Linux Kernel Documentation: Control Group v2 Architecture
The Linux Foundation

Core memory subsystem controls, pressure stall information (PSI), and kernel OOM invocation mechanics.

Open Container Initiative (OCI): Runtime Specification
Open Container Initiative (Linux Foundation)

Industry-standard specification for container execution boundaries and POSIX namespace isolation.

Diagnostic & Architectural FAQ

Q1.Why does GKE Autopilot show 'Insufficient cpu' even though I haven't hit my node's capacity manually?
Your pod spec is fundamentally incompatible with Autopilot's hidden resource tiers. Autopilot abstracts node management, but the underlying nodes have rigid, finite allocatable CPU. When your pod's CPU request plus all existing pod requests on the ready node exceed that node's allocatable capacity, the scheduler correctly emits 'Insufficient cpu'. The real failure is that your request violates Autopilot's CPU/memory ratio bounds or falls below the 250m minimum, creating a pod that no node template can ever satisfy, leaving it Pending indefinitely. This is a spec validation failure you can't see until runtime.
Q2.How long should I wait for the 'node.kubernetes.io/not-ready' taint to clear on GKE Autopilot?
Manually waiting and diagnosing this is a high-risk trap. While a standard provisioning cycle is 60–180 seconds, persistence beyond 5 minutes signals a critical infrastructure failure: quota exhaustion, regional capacity collapse, or a pod spec that Autopilot's admission webhook cannot reconcile. Running 'kubectl describe node' and checking GCP quotas is error-prone and exposes you to security gaps in the diagnostic process. Instead of manual triage, drop the broken manifest into the StackEngine Local Sandbox above to automatically compile a zero-trust fix that resolves the taint and hardens the spec against recurrence.
Q3.Can I use tolerations to force a pod onto a not-ready node in GKE Autopilot?
Adding a toleration for 'node.kubernetes.io/not-ready' is a compliance and security nightmare that creates massive blast radius. While it technically allows scheduling, it forces pods onto unstable, provisioning nodes, causing readiness probe failures and potential data loss. The correct pattern—running multiple replicas with topology spread constraints—is complex to implement manually at scale. Doing this across large mono-repos or in a multi-team CI/CD pipeline is a compliance audit failure waiting to happen. Upgrade to the StackEngine CI/CD Automator (Team Edition) for on-prem, automated pull request generation with Maker-Checker review that enforces these patterns and eliminates manual, insecure overrides.