Fixing Nginx 502 'Upstream Closed Connection While SSL Handshaking' with Self-Signed Certs
Threat/Impact Level: HIGH | Downtime Risk: HIGH | Time to Fix: 5–15 mins
TL;DR
- What broke: Nginx is proxying to an HTTPS upstream (e.g., an internal microservice, Kubernetes sidecar, or legacy app) that presents a self-signed certificate. Nginx's default SSL verification rejects it, closes the connection, and returns a 502 to the client.
- How to fix it: Either disable upstream SSL verification with
proxy_ssl_verify off;(dev/internal only) or supply the self-signed CA cert viaproxy_ssl_trusted_certificateand setproxy_ssl_verify on;(production-grade). - Fast path: Use our Client-Side Sandbox above to auto-refactor this — paste your
locationblock and get the corrected directives without sending your config to a third-party server.
▼ 🔍 Root Cause Breakdown & Kernel Diagnostics [Click to Expand Deep Dive]
The Incident (What Does the Error Mean?)
You'll see this in error.log:
2024/01/15 03:42:17 [error] 1234#1234: *5678 upstream SSL handshaking to 10.0.1.45:8443 while SSL handshaking to upstream,
client: 203.0.113.10, server: api.internal, request: "POST /v1/data HTTP/1.1",
upstream: "https://10.0.1.45:8443/v1/data"And in your access log, the upstream status is 502.
What actually happened: Nginx initiated a TLS ClientHello to the upstream. The upstream responded with its certificate chain. Nginx attempted to verify that chain against its trusted CA store (/etc/ssl/certs/ca-certificates.crt by default). A self-signed cert is not in that store. Nginx sent a TLS alert: certificate unknown, the upstream closed the TCP connection, and Nginx had no response to proxy — hence 502, not 503.
Immediate consequence: 100% of requests to that upstream fail. No fallback, no partial degradation. If this upstream is behind a upstream {} block with multiple servers, the failing server gets marked down after max_fails attempts, potentially cascading to full upstream pool exhaustion.
▼ 🔍 Detailed Impact & Blast Radius Analysis [Click to Expand Deep Dive]
The Attack Vector / Blast Radius
This is a misconfiguration with a dual failure mode:
Operational blast radius: Every request proxied to that backend returns 502. If your health check endpoint also goes through Nginx, your load balancer marks the entire Nginx instance unhealthy. One misconfigured proxy_ssl_verify setting can take down a node entirely.
Security risk of the wrong fix: The instinct is to slam proxy_ssl_verify off; everywhere and move on. That's dangerous in production because:
- You are now vulnerable to machine-in-the-middle attacks on your internal east-west traffic. An attacker with network access between your Nginx instance and the upstream (compromised pod, rogue container, ARP spoofing on a flat internal network) can intercept and decrypt all proxied traffic.
- In regulated environments (PCI-DSS, HIPAA, SOC 2), disabling upstream cert verification on traffic carrying PII or card data is an automatic audit failure.
proxy_ssl_verify off;is not scoped — if you put it in anhttp {}block, it disables verification for every upstream in that Nginx instance.
The correct threat model: Self-signed certs on internal services are acceptable only if Nginx is configured to explicitly trust that specific CA. Blanket verification bypass is a compensating control that requires documented risk acceptance.
How to Fix It (The Solution)
Basic Fix (Dev / Isolated Internal Networks Only)
Add proxy_ssl_verify off; scoped tightly to the specific location block:
location /api/ {
proxy_pass https://10.0.1.45:8443;
+ proxy_ssl_verify off;
proxy_set_header Host $host;
}Do not put this in the http {} or server {} context unless you intend it to apply globally.
Enterprise Best Practice (Production)
Extract the self-signed CA cert from the upstream service and trust it explicitly. This preserves encryption integrity and certificate identity verification.
Step 1 — Pull the upstream's CA cert:
openssl s_client -connect 10.0.1.45:8443 -showcerts </dev/null 2>/dev/null \
| openssl x509 -outform PEM > /etc/nginx/certs/upstream-ca.pemStep 2 — Configure Nginx to trust it:
location /api/ {
proxy_pass https://10.0.1.45:8443;
- # proxy_ssl_verify off; <-- remove or never add this
+ proxy_ssl_verify on;
+ proxy_ssl_trusted_certificate /etc/nginx/certs/upstream-ca.pem;
+ proxy_ssl_verify_depth 2;
+ proxy_ssl_server_name on;
+ proxy_ssl_name internal-service.svc.cluster.local;
proxy_set_header Host $host;
}Why proxy_ssl_server_name on matters: Without it, Nginx sends no SNI extension in the ClientHello. Many modern TLS servers (including Go's net/http and Node.js https) require SNI to select the correct certificate. Missing SNI can cause a different handshake failure even after you fix the CA trust issue.
Why proxy_ssl_verify_depth 2: Self-signed certs are often depth-0 (self-issued root). The default depth of 1 is fine for most cases, but intermediate-signed internal CAs need depth 2+. Set it explicitly rather than relying on defaults.
💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.
(For CI/CD pipeline automation, install via
npm i -g stackengine).
▼ 🔍 Continuous Pipeline Governance & CI/CD Guardrails [Click to Expand Deep Dive]
Prevention in CI/CD
1. Lint Nginx configs in your pipeline:
# In your CI step — catches missing proxy_ssl_verify directives before merge
nginx -t -c /path/to/nginx.confPair with gixy, the Nginx static analyzer:
pip install gixy
gixy /etc/nginx/nginx.confGixy rule http_splitting and custom rules can flag proxy_ssl_verify off in non-dev contexts.
2. OPA/Conftest policy to block proxy_ssl_verify off in production configs:
package nginx.ssl
deny[msg] {
input.proxy_ssl_verify == "off"
input.environment == "production"
msg := "proxy_ssl_verify off is prohibited in production upstream blocks"
}3. Checkov for Kubernetes Ingress / ConfigMap-based Nginx configs:
checkov -d ./k8s/nginx --framework kubernetes4. Automate CA cert rotation: If the upstream service rotates its self-signed cert (cert expiry is the #1 cause of this error re-appearing in production), your upstream-ca.pem goes stale. Use a Kubernetes CronJob or a Vault PKI secrets engine with auto-renewal to keep the trusted CA bundle current. Alert on cert expiry with:
openssl x509 -enddate -noout -in /etc/nginx/certs/upstream-ca.pemSet a monitoring alert 30 days before the notAfter date.
Official Grounding & Normative Specifications
The technical diagnostics, error recovery protocols, and prevention patterns in this specification are grounded in official vendor documentation and standards:
Federal security guidelines for secret exfiltration prevention, automated integrity checks, and peer sign-offs.
Core memory subsystem controls, pressure stall information (PSI), and kernel OOM invocation mechanics.
Industry-standard specification for container execution boundaries and POSIX namespace isolation.