Fixing AKS 'Failed to Get Node Info' During NodePool Scale-Up: Root Cause & Resolution
Threat/Impact Level: HIGH | Downtime Risk: HIGH | Time to Fix: 15–45 mins depending on root cause
TL;DR
- What broke: AKS cluster autoscaler or manual scale-up triggered new node provisioning, but the kubelet on new nodes failed to register —
failed to get node infosurfaces inkube-controller-managerlogs, halting pod scheduling on the new capacity. - How to fix it: Identify whether the failure is Azure VM quota exhaustion, a broken kubelet bootstrap token/RBAC binding, or a corrupted node object in etcd — then execute the targeted remediation below.
- Use our Client-Side Sandbox above to drop your nodepool spec or
kubectl describe nodeoutput and auto-generate the refactored configuration.
▼ 🔍 Root Cause Breakdown & Kernel Diagnostics [Click to Expand Deep Dive]
The Incident (What Does the Error Mean?)
Raw error from kube-controller-manager or kubectl get events:
E0601 14:32:11.847201 1 node_lifecycle_controller.go:1494] "Failed to get node info" err="node \"aks-nodepool1-12345678-vmss000003\" not found" node="aks-nodepool1-12345678-vmss000003"Or from the cluster autoscaler:
scale_up.go:453] Failed to get node info for aks-nodepool1-12345678-vmss000003: nodes "aks-nodepool1-12345678-vmss000003" not foundImmediate consequence: The new VM spun up in the VMSS, the OS booted, but the kubelet never successfully completed node registration with the API server. The node object either never appeared in etcd or appeared in a NotReady state and was immediately evicted. Pending pods remain unscheduled. If this is autoscaler-driven, the autoscaler may enter a backoff loop and stop scaling entirely, silently starving your workload of capacity.
▼ 🔍 Detailed Impact & Blast Radius Analysis [Click to Expand Deep Dive]
The Attack Vector / Blast Radius
This is a cascading availability failure, not a security exploit — but the blast radius is severe:
- Autoscaler deadlock: Autoscaler marks the scale-up as failed, increments its backoff timer. In high-traffic scenarios, you lose the 10-minute autoscaler backoff window entirely while pods pile up in
Pending. - VMSS billing leak: The Azure VM is running and billing you even though it contributes zero capacity. At scale, a stuck nodepool can orphan 10–50 VMs.
- Root causes that compound each other:
- Azure subscription vCPU quota hit mid-scale — VM provisions but ARM returns a partial success; kubelet has no valid identity to register.
system:nodeRBAC bootstrap binding broken — the kubelet TLS bootstrap process cannot obtain a signed certificate, so it cannot authenticate to the API server toGET /api/v1/nodes/{name}.- Corrupted or stale node object — a previous node with the same VMSS instance name exists in a
Terminatingstate, blocking re-registration. - Custom VNet/NSG blocking port 10250 — kubelet API unreachable, node controller cannot collect node info.
How to Fix It
Step 1: Triage in 60 Seconds
# Check for stuck/ghost node objects
kubectl get nodes | grep -E 'NotReady|Unknown'
# Check autoscaler logs for the specific failure class
kubectl logs -n kube-system -l app=cluster-autoscaler --tail=100 | grep -i 'failed\|error\|node info'
# Check Azure quota for the region/SKU
az vm list-usage --location eastus --output table | grep -i 'cores'
# Check kubelet bootstrap RBAC
kubectl get clusterrolebinding kubelet-bootstrap
kubectl get clusterrolebinding system:node-proxierFix A: Stale Node Object Blocking Re-Registration (Most Common)
- # Stale node object holds the name, new kubelet cannot register
- kubectl get node aks-nodepool1-12345678-vmss000003
- # STATUS: Terminating / Unknown for >10 minutes
+ # Force delete the ghost node object
+ kubectl delete node aks-nodepool1-12345678-vmss000003 --grace-period=0 --force
+
+ # Then cordon + drain any surviving VMSS instance and let autoscaler reprovision
+ az vmss delete-instances \
+ --resource-group MC_myRG_myAKS_eastus \
+ --name aks-nodepool1-12345678-vmss \
+ --instance-ids 3Fix B: Broken kubelet Bootstrap RBAC (Enterprise Best Practice)
AKS manages this binding, but custom AAD/RBAC configurations or manual cluster modifications can break it.
- # Missing or malformed bootstrap binding
- kubectl get clusterrolebinding kubelet-bootstrap
- # Error: clusterrolebindings.rbac.authorization.k8s.io "kubelet-bootstrap" not found
+ # Restore the bootstrap binding
+ kubectl create clusterrolebinding kubelet-bootstrap \
+ --clusterrole=system:node-bootstrapper \
+ --group=system:bootstrappers
+
+ # Verify node-autoapprove binding exists
+ kubectl get clusterrolebinding system:node-autoapprove-bootstrap
+ # If missing:
+ kubectl apply -f https://raw.githubusercontent.com/Azure/AKS/master/examples/bootstrap/node-autoapprove.yamlFix C: Azure vCPU Quota Exhaustion
- # Nodepool scale-up requesting Standard_D4s_v3 (4 vCPUs x 10 nodes = 40 vCPUs)
- # Current quota: 32 vCPUs — silent partial provision, 2 VMs boot without valid ARM identity
+ # Request quota increase (takes 1–24 hrs via support ticket)
+ az support tickets create \
+ --ticket-name "vCPU quota increase eastus" \
+ --title "Increase Standard DSv3 quota" \
+ --contact-country "US" \
+ --contact-email "ops@company.com"
+
+ # Immediate workaround: switch nodepool to a SKU with available quota
+ az aks nodepool add \
+ --resource-group myRG \
+ --cluster-name myAKS \
+ --name fallbackpool \
+ --node-vm-size Standard_D4s_v4 \
+ --node-count 5Fix D: NSG Blocking Kubelet Port 10250
- # NSG rule denying inbound 10250 from API server subnet to node subnet
- {
- "name": "DenyAll",
- "priority": 100,
- "direction": "Inbound",
- "access": "Deny",
- "protocol": "*",
- "sourceAddressPrefix": "*",
- "destinationPortRange": "*"
- }
+ # Add explicit allow rule for AKS control plane to reach kubelet
+ {
+ "name": "AllowAKSControlPlaneKubelet",
+ "priority": 90,
+ "direction": "Inbound",
+ "access": "Allow",
+ "protocol": "Tcp",
+ "sourceAddressPrefix": "AzureCloud.EastUS",
+ "destinationPortRange": "10250"
+ }💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.
(For CI/CD pipeline automation, install via
npm i -g stackengine).
▼ 🔍 Continuous Pipeline Governance & CI/CD Guardrails [Click to Expand Deep Dive]
Prevention in CI/CD
1. Pre-Scale Quota Gate in your Pipeline
# Azure DevOps / GitHub Actions step before any nodepool scale operation
- name: Check vCPU Quota Before Scale
run: |
AVAILABLE=$(az vm list-usage --location $REGION \
--query "[?localName=='Standard DSv3 Family vCPUs'].currentValue" -o tsv)
LIMIT=$(az vm list-usage --location $REGION \
--query "[?localName=='Standard DSv3 Family vCPUs'].limit" -o tsv)
HEADROOM=$((LIMIT - AVAILABLE))
REQUIRED=$((NODE_COUNT * VCPUS_PER_NODE))
if [ $HEADROOM -lt $REQUIRED ]; then
echo "FATAL: Insufficient vCPU quota. Headroom: $HEADROOM, Required: $REQUIRED"
exit 1
fi2. OPA/Gatekeeper Policy — Enforce NodePool SKU Allowlist
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sAllowedNodeSKUs
metadata:
name: enforce-approved-vm-skus
spec:
match:
kinds:
- apiGroups: ["*"]
kinds: ["Node"]
parameters:
allowedSKUs:
- "Standard_D4s_v3"
- "Standard_D8s_v3"
- "Standard_D4s_v4"3. Checkov / Terraform Lint for NSG Rules
# Add to your Terraform CI pipeline
checkov -d ./infra/aks \
--check CKV_AZURE_77 \
--check CKV_AZURE_9 \
--compact
# CKV_AZURE_77: Ensure AKS node pools use authorized IP ranges
# CKV_AZURE_9: Ensure NSG does not have unrestricted inbound deny-all before AKS rules4. Alert on Node Registration Latency
# Prometheus alert — fire if a node stays NotReady > 5 minutes post-provision
- alert: AKSNodeRegistrationStuck
expr: |
(kube_node_status_condition{condition="Ready",status="false"} == 1)
and
(time() - kube_node_created > 300)
for: 2m
labels:
severity: critical
annotations:
summary: "Node {{ $labels.node }} stuck in NotReady > 5min post-provision"
runbook: "https://your-wiki/aks-node-registration-runbook"Official Grounding & Normative Specifications
The technical diagnostics, error recovery protocols, and prevention patterns in this specification are grounded in official vendor documentation and standards:
Official runtime configuration documentation governing lock acquisition timeouts, deadlock detection cycles, and wait logging.
Federal security guidelines for secret exfiltration prevention, automated integrity checks, and peer sign-offs.
Core memory subsystem controls, pressure stall information (PSI), and kernel OOM invocation mechanics.
Industry-standard specification for container execution boundaries and POSIX namespace isolation.