Istio Sidecar Mutating Webhook Timeout

Threat/Impact Level: HIGH | Downtime Risk: HIGH | Time to Fix: 15–30 mins

TL;DR

• What broke: Kubernetes could not complete the mutating admission webhook call to istiod within the configured timeout, so the pod was rejected or admitted without a sidecar — silently breaking it from the mesh.
• How to fix it: Increase timeoutSeconds on the MutatingWebhookConfiguration, ensure istiod has sufficient CPU/memory headroom, and set failurePolicy: Fail only after confirming istiod HA.
• Use our Client-Side Sandbox above to paste your MutatingWebhookConfiguration and auto-refactor it with corrected timeout, failure policy, and namespace selectors.

Warning  FailedCreate  admission webhook "istio-sidecar-injector.istio.io" 
failed calling webhook: Post 
"https://istiod.istio-system.svc:443/inject?timeout=10s": 
context deadline exceeded

Error from server (InternalError): Internal error occurred: 
failed calling webhook "istio-sidecar-injector.istio.io": 
failed to call webhook: Post 
"https://istiod.istio-system.svc:443/inject": 
net/http: request canceled (Client.Timeout exceeded)

How to Fix It (The Solution)

Root Cause Checklist — Run These First

# 1. Is istiod running and ready?
kubectl get pods -n istio-system -l app=istiod

# 2. Check istiod resource pressure
kubectl top pod -n istio-system -l app=istiod

# 3. Inspect the webhook config timeout
kubectl get mutatingwebhookconfiguration istio-sidecar-injector \
  -o jsonpath='{.webhooks[*].timeoutSeconds}'

# 4. Check for NetworkPolicy blocking 443 to istiod
kubectl get networkpolicy -A

# 5. Recent istiod errors
kubectl logs -n istio-system -l app=istiod --tail=100 | grep -i error

Basic Fix — Patch the Webhook Timeout

The Kubernetes API server default webhook timeout is 10 seconds. Under load, istiod needs more headroom.

# kubectl edit mutatingwebhookconfiguration istio-sidecar-injector

  webhooks:
  - name: istio-sidecar-injector.istio.io
-   timeoutSeconds: 10
+   timeoutSeconds: 25
-   failurePolicy: Ignore
+   failurePolicy: Fail
    admissionReviewVersions: ["v1", "v1beta1"]
    clientConfig:
      service:
        name: istiod
        namespace: istio-system
        path: "/inject"
        port: 443

⚠️ Only set failurePolicy: Fail after confirming istiod is highly available (≥2 replicas with a PodDisruptionBudget). Otherwise you trade a silent mesh bypass for a hard scheduling outage.

Enterprise Best Practice — istiod HA + Resource Tuning

1. Scale istiod and set a PDB:

# istiod Deployment (via IstioOperator or Helm values)
  spec:
    components:
      pilot:
        k8s:
-         replicaCount: 1
+         replicaCount: 3
+         podDisruptionBudget:
+           minAvailable: 2
          resources:
            requests:
-             cpu: 100m
-             memory: 128Mi
+             cpu: 500m
+             memory: 512Mi
            limits:
+             cpu: 2000m
+             memory: 2Gi
          hpaSpec:
+           minReplicas: 2
+           maxReplicas: 5
+           metrics:
+           - type: Resource
+             resource:
+               name: cpu
+               target:
+                 type: Utilization
+                 averageUtilization: 60

2. Tighten namespace selector to avoid injecting system namespaces (reduces webhook call volume):

  webhooks:
  - name: istio-sidecar-injector.istio.io
    namespaceSelector:
      matchExpressions:
-     - key: istio-injection
-       operator: In
-       values: ["enabled"]
+     - key: istio-injection
+       operator: In
+       values: ["enabled"]
+     - key: kubernetes.io/metadata.name
+       operator: NotIn
+       values: ["kube-system", "kube-public", "istio-system", "cert-manager"]

3. If a NetworkPolicy is blocking the API server → istiod path:

+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: allow-apiserver-to-istiod-webhook
+  namespace: istio-system
+spec:
+  podSelector:
+    matchLabels:
+      app: istiod
+  ingress:
+  - ports:
+    - port: 443
+      protocol: TCP
+    - port: 15017
+      protocol: TCP

Note: Port 15017 is the istiod webhook port in Istio ≥1.10. Port 443 is the Kubernetes service port that maps to it. Verify with kubectl get svc istiod -n istio-system.

💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.

(For CI/CD pipeline automation, install via npm i -g stackengine).

# ConstraintTemplate: deny webhook timeoutSeconds < 20
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: webhooktimeoutminimum
spec:
  crd:
    spec:
      names:
        kind: WebhookTimeoutMinimum
  targets:
  - target: admission.k8s.gatekeeper.sh
    rego: |
      package webhooktimeoutminimum
      violation[{"msg": msg}] {
        webhook := input.review.object.webhooks[_]
        webhook.timeoutSeconds < 20
        msg := sprintf("Webhook '%v' timeoutSeconds must be >= 20, got %v",
          [webhook.name, webhook.timeoutSeconds])
      }

# Add to your CI pipeline (GitHub Actions, GitLab CI, etc.)
checkov -d ./istio/manifests \
  --check CKV_K8S_35 \
  --framework kubernetes \
  --compact

# Pin and validate istiod values on every Helm upgrade in CI
helm upgrade istio-base istio/base \
  --dry-run \
  --values ./prod-values.yaml \
  | kubeval --strict -

# Alert if any running pod in an injection-enabled namespace lacks the sidecar
- alert: IstioPodMissingProxy
  expr: |
    kube_pod_container_info{container!="istio-proxy"}
    unless on(pod, namespace)
    kube_pod_container_info{container="istio-proxy"}
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Pod {{ $labels.pod }} in {{ $labels.namespace }} is running without Istio sidecar"