How to Fix Nginx Ingress 502 Bad Gateway: Backend Service Port Mismatch Debugging Guide
Threat/Impact Level: HIGH | Downtime Risk: HIGH | Time to Fix: 5–15 mins
TL;DR
- What broke: Your Nginx Ingress
backend.service.portreferences a port number or name that does not exist on the target Kubernetes Service, so the ingress controller upstream resolves to nothing and returns 502. - How to fix it: Align
spec.rules[].http.paths[].backend.service.port.number(or.name) in the Ingress exactly withspec.ports[].port(or.name) in the corresponding Service manifest. - Fast path: Use our Client-Side Sandbox above to auto-refactor this — paste both your Ingress and Service YAML and get a corrected diff without sending secrets anywhere.
▼ 🔍 Root Cause Breakdown & Kernel Diagnostics [Click to Expand Deep Dive]
The Incident (What Does the Error Mean?)
Your pod logs are clean. Your app container is healthy. But every request hitting the Ingress returns:
HTTP/1.1 502 Bad Gateway
Server: nginxAnd in the nginx-ingress-controller pod logs:
[error] 2048#2048: *1983 connect() failed (111: Connection refused) while connecting to upstream,
client: 10.0.0.1, server: api.example.com, request: "GET /api/v1/health HTTP/1.1",
upstream: "http://10.96.45.12:3001/api/v1/health", host: "api.example.com"or the more telling variant:
upstream connect error or disconnect/reset before headers. reset reason: connection failureImmediate consequence: 100% of traffic routed through this Ingress rule is failing. No requests reach your backend pods. The upstream IP resolves correctly (kube-proxy is fine), but the port is wrong — the controller is hammering a port that nothing is listening on.
▼ 🔍 Detailed Impact & Blast Radius Analysis [Click to Expand Deep Dive]
The Attack Vector / Blast Radius
This is not a security exploit — but the blast radius in production is total. Every downstream consumer of this service (frontend, mobile clients, partner API integrations, internal microservices) receives 502s. If this is behind a load balancer with health checks, the target group will mark the node unhealthy, triggering cascading deregistration.
Why this happens more than it should:
- Helm chart values override
service.portbut the Ingress template is hardcoded to a default (e.g.,80vs8080). - A developer renames a Service port from
httptowebbut doesn't update the Ingressport.namereference. - Kubernetes 1.19+ Ingress v1 API changed
servicePort(IntOrString) toservice.port.number/service.port.name— copy-pasted v1beta1 manifests silently misconfigure. - Multi-port Services where the wrong index is targeted.
The nginx-ingress-controller does not fail loudly at apply time. kubectl apply succeeds with exit 0. The misconfiguration only surfaces under live traffic. This makes it a silent production killer.
How to Fix It
Basic Fix: Align the Port in the Ingress Backend
Verify what port your Service actually exposes:
kubectl get svc my-api-service -n production -o yaml | grep -A 10 'ports:'Expected output:
ports:
- name: http
port: 8080
targetPort: 3000
protocol: TCPNow check your Ingress:
kubectl get ingress my-api-ingress -n production -o yaml | grep -A 10 'backend:'Broken output:
backend:
service:
name: my-api-service
port:
number: 80 # <-- WRONG. Service exposes 8080.The diff fix:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-api-ingress
namespace: production
spec:
rules:
- host: api.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: my-api-service
port:
- number: 80
+ number: 8080Or using port name reference (preferred — resilient to port number changes):
service:
name: my-api-service
port:
- number: 80
+ name: httpApply and verify:
kubectl apply -f ingress.yaml
kubectl describe ingress my-api-ingress -n production
# Look for: "Endpoints: 10.x.x.x:8080" — if it shows "<none>", port still mismatched.Enterprise Best Practice: Named Ports + Admission Validation
Hardcoded port numbers are a maintenance liability. Use named ports end-to-end and enforce consistency with an admission webhook or OPA policy.
Service manifest — always name your ports:
apiVersion: v1
kind: Service
metadata:
name: my-api-service
namespace: production
spec:
selector:
app: my-api
ports:
- - port: 8080
- targetPort: 3000
+ - name: http
+ port: 8080
+ targetPort: http # matches containerPort name in DeploymentDeployment — name your containerPorts:
containers:
- name: my-api
image: my-api:v2.1.0
ports:
- - containerPort: 3000
+ - name: http
+ containerPort: 3000Ingress — reference by name, never by magic number:
backend:
service:
name: my-api-service
port:
- number: 80
+ name: httpNow a port number change in the Service does not break the Ingress. The name http is the contract.
💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.
(For CI/CD pipeline automation, install via
npm i -g stackengine).
▼ 🔍 Continuous Pipeline Governance & CI/CD Guardrails [Click to Expand Deep Dive]
Prevention in CI/CD
1. Kubeval / Kubeconform in PR Pipeline
Catch schema violations before merge:
kubeconform -strict -kubernetes-version 1.27.0 ./k8s/This won't catch semantic mismatches, but it eliminates copy-paste v1beta1 → v1 API errors.
2. Conftest + OPA Policy: Enforce Named Ports
Write a Rego policy that fails any Ingress referencing a backend port by number:
# policy/ingress-named-ports.rego
package main
deny[msg] {
input.kind == "Ingress"
path := input.spec.rules[_].http.paths[_]
port := path.backend.service.port
not port.name
msg := sprintf(
"Ingress '%v': backend service port must use 'name', not 'number'. Got port number: %v",
[input.metadata.name, port.number]
)
}conftest test ./k8s/ingress.yaml --policy ./policy/3. Pluto: Detect Deprecated API Versions
pluto detect-files -d ./k8s/ --target-versions k8s=v1.27.0Flags any networking.k8s.io/v1beta1 Ingress manifests still in your repo.
4. Helm Chart Validation
If you're generating Ingress via Helm, add a _helpers.tpl guard:
{{- if and .Values.service.port (not .Values.service.portName) }}
{{- fail "values.service.portName is required. Do not use bare port numbers in Ingress backend." }}
{{- end }}5. Post-Deploy Smoke Test
Add a 30-second post-deploy curl check in your CD pipeline:
STATUS=$(curl -o /dev/null -s -w "%{http_code}" https://api.example.com/health)
if [ "$STATUS" != "200" ]; then
echo "DEPLOY FAILED: Got HTTP $STATUS — possible Ingress port mismatch. Rolling back."
kubectl rollout undo deployment/my-api -n production
exit 1
fiThis catches 502s within seconds of a bad deploy before they hit your SLA window.
Official Grounding & Normative Specifications
The technical diagnostics, error recovery protocols, and prevention patterns in this specification are grounded in official vendor documentation and standards:
Official runtime configuration documentation governing lock acquisition timeouts, deadlock detection cycles, and wait logging.
Core concepts governing declarative infrastructure lifecycle, state lock contention, and provider graph synthesis.