EKS ALB Controller Webhook Cert Renewer

Threat/Impact Level: CRITICAL | Downtime Risk: HIGH | Time to Fix: 10–20 mins

TL;DR

• What broke: The TLS certificate backing the aws-load-balancer-webhook-service expired. Kubernetes rejects all webhook calls, blocking every Ingress and Service of type LoadBalancer from being created or updated.
• How to fix it: Delete the expired aws-load-balancer-tls secret, re-run cert generation (via cert-manager or manual openssl), and restart the controller pod to force re-registration.
• Fast path: Use our Client-Side Sandbox above to paste your kubectl describe mutatingwebhookconfiguration output and auto-generate the exact rotation commands for your cluster.

Error from server (InternalError): error when creating "ingress.yaml":
  Internal error occurred: failed calling webhook
  "mwebhook.elbv2.k8s.aws": failed to call webhook:
  Post "https://aws-load-balancer-webhook-service.kube-system.svc:443/mutate-v1-service?timeout=10s":
  x509: certificate has expired or is not yet valid:
  current time 2024-11-01T10:22:31Z is after 2024-06-15T00:00:00Z

tls: no certificates configured
WebhookServer: unable to start server: tls handshake error

How to Fix It

Step 1 — Confirm the expiry

# Check the certificate expiry in the secret
kubectl get secret aws-load-balancer-tls \
  -n kube-system \
  -o jsonpath='{.data.tls\.crt}' \
  | base64 -d \
  | openssl x509 -noout -dates

# Check the webhook config CA bundle
kubectl get mutatingwebhookconfiguration \
  aws-load-balancer-mutating-webhook-configuration \
  -o jsonpath='{.webhooks[0].clientConfig.caBundle}' \
  | base64 -d \
  | openssl x509 -noout -dates

Basic Fix — Manual Certificate Rotation

Use this if you are not running cert-manager.

# 1. Generate a new self-signed cert (10-year validity; adjust to policy)
OPENSSL_SAN="dns:aws-load-balancer-webhook-service.kube-system.svc,\
dns:aws-load-balancer-webhook-service.kube-system.svc.cluster.local"

openssl req -x509 -newkey rsa:4096 -nodes \
  -keyout tls.key -out tls.crt \
  -days 3650 \
  -subj "/CN=aws-load-balancer-webhook-service.kube-system.svc" \
  -addext "subjectAltName=${OPENSSL_SAN}"

# 2. Delete and recreate the secret
kubectl delete secret aws-load-balancer-tls -n kube-system
kubectl create secret tls aws-load-balancer-tls \
  -n kube-system \
  --cert=tls.crt \
  --key=tls.key

# 3. Patch the webhook CA bundle
CA_BUNDLE=$(base64 -w0 < tls.crt)
kubectl patch mutatingwebhookconfiguration \
  aws-load-balancer-mutating-webhook-configuration \
  --type='json' \
  -p="[{\"op\":\"replace\",\"path\":\"/webhooks/0/clientConfig/caBundle\",\"value\":\"${CA_BUNDLE}\"}]"

kubectl patch validatingwebhookconfiguration \
  aws-load-balancer-validating-webhook-configuration \
  --type='json' \
  -p="[{\"op\":\"replace\",\"path\":\"/webhooks/0/clientConfig/caBundle\",\"value\":\"${CA_BUNDLE}\"}]"

# 4. Restart the controller
kubectl rollout restart deployment aws-load-balancer-controller -n kube-system
kubectl rollout status deployment aws-load-balancer-controller -n kube-system

Enterprise Best Practice — cert-manager Automated Rotation

This is the only acceptable long-term solution. Manual cert rotation is a toil trap.

# helm/values-aws-lbc.yaml

- # No cert-manager integration; using manually managed secret
- webhookTLS:
-   secretName: aws-load-balancer-tls

+ # cert-manager issues and auto-rotates the webhook certificate
+ enableCertManager: true
+
+ # Ensure cert-manager is installed before this chart
+ # helm repo add jetstack https://charts.jetstack.io
+ # helm install cert-manager jetstack/cert-manager \
+ #   --namespace cert-manager --create-namespace \
+ #   --set installCRDs=true

# cert-manager Certificate resource (if managing manually via CRD)

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: aws-load-balancer-serving-cert
  namespace: kube-system
spec:
  dnsNames:
-   - aws-load-balancer-webhook-service
+   - aws-load-balancer-webhook-service.kube-system.svc
+   - aws-load-balancer-webhook-service.kube-system.svc.cluster.local
  issuerRef:
    kind: Issuer
    name: aws-load-balancer-selfsigned-issuer
- duration: 8760h   # 1 year, manual rotation required
+ duration: 8760h
+ renewBefore: 720h  # Renew 30 days before expiry — cert-manager handles this automatically
  secretName: aws-load-balancer-tls

With cert-manager and renewBefore set, rotation is fully automated. The controller picks up the new secret on the next reconcile cycle without a pod restart.

💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.

(For CI/CD pipeline automation, install via npm i -g stackengine).

# prometheus-rules.yaml
groups:
  - name: eks-webhook-cert
    rules:
      - alert: EKSWebhookCertExpiringSoon
        expr: |
          (certmanager_certificate_expiration_timestamp_seconds
            {name="aws-load-balancer-serving-cert"} - time()) / 86400 < 30
        for: 1h
        labels:
          severity: critical
        annotations:
          summary: "AWS LBC webhook cert expires in < 30 days"

# opa/lbc-certmanager-required.rego
package lbc

deny[msg] {
  input.review.object.kind == "HelmRelease"
  input.review.object.spec.chart.spec.chart == "aws-load-balancer-controller"
  not input.review.object.spec.values.enableCertManager == true
  msg := "AWS LBC HelmRelease must have enableCertManager: true"
}

# .github/workflows/checkov.yaml (relevant step)
- name: Scan Helm values for LBC misconfig
  run: |
    checkov -d ./helm \
      --framework kubernetes \
      --check CKV_K8S_WEBHOOK_CERT_MANAGED

# argocd-application.yaml
spec:
  source:
    helm:
      values: |
        enableCertManager: true
  syncPolicy:
    syncOptions:
      - RespectIgnoreDifferences=true
    retry:
      limit: 3