How to Fix Nginx Ingress 502 Bad Gateway: Backend Service Port Mismatch Debugging Guide

Threat/Impact Level: HIGH | Downtime Risk: HIGH | Time to Fix: 5–15 mins

TL;DR

• What broke: Your Nginx Ingress backend.service.port references a port number or name that does not exist on the target Kubernetes Service, so the ingress controller upstream resolves to nothing and returns 502.
• How to fix it: Align spec.rules[].http.paths[].backend.service.port.number (or .name) in the Ingress exactly with spec.ports[].port (or .name) in the corresponding Service manifest.
• Fast path: Use our Client-Side Sandbox below to auto-refactor this — paste both your Ingress and Service YAML and get a corrected diff without sending secrets anywhere.

---

The Incident (What Does the Error Mean?)

Your pod logs are clean. Your app container is healthy. But every request hitting the Ingress returns:

HTTP/1.1 502 Bad Gateway
Server: nginx

And in the nginx-ingress-controller pod logs:

[error] 2048#2048: *1983 connect() failed (111: Connection refused) while connecting to upstream,
client: 10.0.0.1, server: api.example.com, request: "GET /api/v1/health HTTP/1.1",
upstream: "http://10.96.45.12:3001/api/v1/health", host: "api.example.com"

or the more telling variant:

upstream connect error or disconnect/reset before headers. reset reason: connection failure

Immediate consequence: 100% of traffic routed through this Ingress rule is failing. No requests reach your backend pods. The upstream IP resolves correctly (kube-proxy is fine), but the port is wrong — the controller is hammering a port that nothing is listening on.

---

The Attack Vector / Blast Radius

This is not a security exploit — but the blast radius in production is total. Every downstream consumer of this service (frontend, mobile clients, partner API integrations, internal microservices) receives 502s. If this is behind a load balancer with health checks, the target group will mark the node unhealthy, triggering cascading deregistration.

Why this happens more than it should:

• Helm chart values override service.port but the Ingress template is hardcoded to a default (e.g., 80 vs 8080).
• A developer renames a Service port from http to web but doesn't update the Ingress port.name reference.
• Kubernetes 1.19+ Ingress v1 API changed servicePort (IntOrString) to service.port.number / service.port.name — copy-pasted v1beta1 manifests silently misconfigure.
• Multi-port Services where the wrong index is targeted.

The nginx-ingress-controller does not fail loudly at apply time. kubectl apply succeeds with exit 0. The misconfiguration only surfaces under live traffic. This makes it a silent production killer.

---

How to Fix It

Basic Fix: Align the Port in the Ingress Backend

Verify what port your Service actually exposes:

kubectl get svc my-api-service -n production -o yaml | grep -A 10 'ports:'

Expected output:

ports:
  - name: http
    port: 8080
    targetPort: 3000
    protocol: TCP

Now check your Ingress:

kubectl get ingress my-api-ingress -n production -o yaml | grep -A 10 'backend:'

Broken output:

backend:
  service:
    name: my-api-service
    port:
      number: 80   # <-- WRONG. Service exposes 8080.

The diff fix:

 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
   name: my-api-ingress
   namespace: production
 spec:
   rules:
     - host: api.example.com
       http:
         paths:
           - path: /
             pathType: Prefix
             backend:
               service:
                 name: my-api-service
                 port:
-                  number: 80
+                  number: 8080

Or using port name reference (preferred — resilient to port number changes):

               service:
                 name: my-api-service
                 port:
-                  number: 80
+                  name: http

Apply and verify:

kubectl apply -f ingress.yaml
kubectl describe ingress my-api-ingress -n production
# Look for: "Endpoints: 10.x.x.x:8080" — if it shows "<none>", port still mismatched.

---

Enterprise Best Practice: Named Ports + Admission Validation

Hardcoded port numbers are a maintenance liability. Use named ports end-to-end and enforce consistency with an admission webhook or OPA policy.

Service manifest — always name your ports:

 apiVersion: v1
 kind: Service
 metadata:
   name: my-api-service
   namespace: production
 spec:
   selector:
     app: my-api
   ports:
-    - port: 8080
-      targetPort: 3000
+    - name: http
+      port: 8080
+      targetPort: http   # matches containerPort name in Deployment

Deployment — name your containerPorts:

 containers:
   - name: my-api
     image: my-api:v2.1.0
     ports:
-      - containerPort: 3000
+      - name: http
+        containerPort: 3000

Ingress — reference by name, never by magic number:

             backend:
               service:
                 name: my-api-service
                 port:
-                  number: 80
+                  name: http

Now a port number change in the Service does not break the Ingress. The name http is the contract.

---

💡 Tired of pasting proprietary configs into ChatGPT? Generic AI tools log your company's ARNs, DB strings, and private keys. StackEngine is a zero-backend, pure Client-Side WASM utility. Drop your failing config into the sandbox above. We redact your secrets locally in the browser and auto-generate the refactored code using your own API key.

---

Prevention in CI/CD

1. Kubeval / Kubeconform in PR Pipeline

Catch schema violations before merge:

kubeconform -strict -kubernetes-version 1.27.0 ./k8s/

This won't catch semantic mismatches, but it eliminates copy-paste v1beta1 → v1 API errors.

2. Conftest + OPA Policy: Enforce Named Ports

Write a Rego policy that fails any Ingress referencing a backend port by number:

# policy/ingress-named-ports.rego
package main

deny[msg] {
  input.kind == "Ingress"
  path := input.spec.rules[_].http.paths[_]
  port := path.backend.service.port
  not port.name
  msg := sprintf(
    "Ingress '%v': backend service port must use 'name', not 'number'. Got port number: %v",
    [input.metadata.name, port.number]
  )
}

conftest test ./k8s/ingress.yaml --policy ./policy/

3. Pluto: Detect Deprecated API Versions

pluto detect-files -d ./k8s/ --target-versions k8s=v1.27.0

Flags any networking.k8s.io/v1beta1 Ingress manifests still in your repo.

4. Helm Chart Validation

If you're generating Ingress via Helm, add a _helpers.tpl guard:

{{- if and .Values.service.port (not .Values.service.portName) }}
{{- fail "values.service.portName is required. Do not use bare port numbers in Ingress backend." }}
{{- end }}

5. Post-Deploy Smoke Test

Add a 30-second post-deploy curl check in your CD pipeline:

STATUS=$(curl -o /dev/null -s -w "%{http_code}" https://api.example.com/health)
if [ "$STATUS" != "200" ]; then
  echo "DEPLOY FAILED: Got HTTP $STATUS — possible Ingress port mismatch. Rolling back."
  kubectl rollout undo deployment/my-api -n production
  exit 1
fi

This catches 502s within seconds of a bad deploy before they hit your SLA window.