CoreDNS High Load Resolution Error Handler

Threat/Impact Level: HIGH | Downtime Risk: HIGH | Time to Fix: 15–30 mins

TL;DR

• What broke: CoreDNS pods are OOM-killed or CPU-throttled under query surge, causing the ClusterIP 10.96.0.10:53 to refuse connections cluster-wide.
• How to fix it: Scale CoreDNS replicas, tune resource limits, enable DNS caching, and reduce ndots on workload pods to slash upstream query volume.
• Shortcut: Use our Client-Side Sandbox above to drop your CoreDNS ConfigMap and Deployment YAML — it auto-refactors the config locally without sending your data anywhere.

dial tcp 10.96.0.10:53: connect: connection refused
EOF
context deadline exceeded

How to Fix It

Basic Fix — Scale Replicas and Loosen Resource Limits

# coredns Deployment in kube-system
 spec:
   replicas:
-    1
+    3
   containers:
   - name: coredns
     resources:
       limits:
-        cpu: 100m
-        memory: 70Mi
+        cpu: 500m
+        memory: 256Mi
       requests:
-        cpu: 100m
-        memory: 70Mi
+        cpu: 200m
+        memory: 128Mi

Also add a PodDisruptionBudget immediately:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: coredns-pdb
  namespace: kube-system
spec:
  minAvailable: 2
  selector:
    matchLabels:
      k8s-app: kube-dns

Enterprise Best Practice — Cache Tuning + ndots Reduction + HPA

1. CoreDNS ConfigMap — enable aggressive caching and health checks:

 apiVersion: v1
 kind: ConfigMap
 metadata:
   name: coredns
   namespace: kube-system
 data:
   Corefile: |
     .:53 {
         errors
         health {
+            lameduck 5s
         }
+        ready
         kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
+          ttl 30
         }
+        cache 30 {
+            success 9984 30
+            denial 9984 5
+        }
         forward . /etc/resolv.conf {
+            max_concurrent 1000
+            prefer_udp
         }
-        cache 30
         loop
         reload
         loadbalance
     }

2. Reduce ndots on workload pods to stop the query amplification storm:

 # In your application Deployment spec.template.spec
   dnsConfig:
+    options:
+      - name: ndots
+        value: "2"
+      - name: single-request-reopen

ndots: 2 means redis.default.svc.cluster.local resolves in 1 query instead of 6. This is the single highest-impact change for QPS reduction.

3. Add HPA for CoreDNS (Kubernetes 1.23+):

+apiVersion: autoscaling/v2
+kind: HorizontalPodAutoscaler
+metadata:
+  name: coredns
+  namespace: kube-system
+spec:
+  scaleTargetRef:
+    apiVersion: apps/v1
+    kind: Deployment
+    name: coredns
+  minReplicas: 2
+  maxReplicas: 10
+  metrics:
+  - type: Resource
+    resource:
+      name: cpu
+      target:
+        type: Utilization
+        averageUtilization: 70

4. Spread CoreDNS pods across nodes with topology constraints:

+      topologySpreadConstraints:
+      - maxSkew: 1
+        topologyKey: kubernetes.io/hostname
+        whenUnsatisfiable: DoNotSchedule
+        labelSelector:
+          matchLabels:
+            k8s-app: kube-dns

💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.

(For CI/CD pipeline automation, install via npm i -g stackengine).

package coredns

violation[{"msg": msg}] {
  input.review.object.metadata.name == "coredns"
  input.review.object.spec.replicas < 2
  msg := "CoreDNS must run at least 2 replicas"
}

# .checkov/custom_checks/ndots_check.yaml
id: CKV_CUSTOM_NDOTS
name: "Ensure ndots is set to 2 or less"
category: NETWORKING
definition:
  and:
    - cond_type: attribute
      resource_types: [kubernetes_deployment]
      attribute: spec.template.spec.dnsConfig.options
      operator: contains
      value: "ndots"

# Run dnsperf against kube-dns ClusterIP from within cluster
kubectl run dnsperf --image=guessi/dnsperf:latest --rm -it -- \
  dnsperf -s 10.96.0.10 -d /queries.txt -l 30 -c 20 -Q 5000

# Prometheus alert
- alert: CoreDNSHighLatency
  expr: histogram_quantile(0.99, rate(coredns_dns_request_duration_seconds_bucket[5m])) > 0.05
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "CoreDNS p99 latency > 50ms — scale now before connection refused"