Nginx Upstream SSL Handshake 110 Timeout

The Incident (What Does This Error Mean?)

You will see this in /var/log/nginx/error.log:

2024/01/15 03:42:17 [error] 1234#1234: *5678 upstream timed out (110: Connection timed out)
  while SSL handshaking to upstream, client: 10.0.1.45, server: api.internal,
  request: "POST /v2/data HTTP/1.1", upstream: "https://10.0.2.100:8443/v2/data",
  host: "api.internal"

Immediate consequence: Every request proxied to that upstream returns a 502 Bad Gateway to the client. The TLS handshake — TCP connect, ClientHello, ServerHello, certificate exchange, key agreement — did not complete within the timeout window. Nginx terminated the socket. Your upstream never processed the request.

This is not a network packet loss issue by default. The TCP connection typically succeeds. The handshake itself stalls — meaning the problem lives in one of: certificate chain resolution, OCSP stapling, cipher negotiation mismatch, overloaded upstream TLS terminator, or a misconfigured proxy_ssl_* directive on the Nginx side.

The Attack Vector / Blast Radius

This is a availability and cascading failure scenario, not a direct exploit vector — but the blast radius is severe:

1. Connection pool exhaustion: Nginx worker processes hold open sockets waiting for the handshake to complete. Under load, this saturates worker_connections. New requests queue, then fail. The entire reverse proxy layer goes dark.

2. Retry storms: Upstream clients (mobile apps, other services) retry on 502. Each retry spawns a new connection attempt. Each new attempt hits the same stalled handshake. You get an exponential pile-up.

3. Hidden certificate expiry: A common trigger is an upstream certificate that has expired or has a broken intermediate chain. proxy_ssl_verify on causes Nginx to attempt OCSP/CRL validation against an external CA endpoint. If that CA endpoint is unreachable from your VPC (common in air-gapped or strict egress environments), every handshake hangs until timeout. This means a cert rotation event silently kills your proxy tier.

4. Session reuse disabled: Without proxy_ssl_session_reuse on, every proxied request performs a full TLS handshake. At high RPS against a slow upstream TLS stack (e.g., Java-based services with expensive RSA key exchange), the upstream TLS thread pool saturates and stops responding to new ClientHellos in time.

Prevention in CI/CD

1. Validate Nginx Configs in the Pipeline Before Deploy

# In your GitHub Actions / GitLab CI deploy step
nginx -t -c /path/to/nginx.conf
# Non-zero exit code blocks the deploy

This catches syntax errors but not semantic SSL misconfigs. You need the steps below for that.

2. Certificate Chain Validation in CI

Add a pre-deploy step that validates the upstream certificate chain is reachable and valid from within your deployment environment:

# Run this from inside your cluster/VPC during CI
openssl s_client \
  -connect 10.0.2.100:8443 \
  -CAfile /etc/ssl/private/internal-ca-chain.pem \
  -verify_return_error \
  -brief < /dev/null

# Exit code 0 = chain valid and handshake completes
# Non-zero = your Nginx WILL hit SSL handshake timeouts in prod

3. Checkov / OPA Policy — Enforce `proxy_ssl_session_reuse`

For teams using Nginx config linting via gixy or custom OPA policies:

# opa/nginx_ssl_policy.rego
package nginx.ssl

deny[msg] {
  input.proxy_ssl_session_reuse == "off"
  msg := "proxy_ssl_session_reuse must be 'on' for upstream SSL blocks to prevent handshake exhaustion"
}

deny[msg] {
  not input.proxy_ssl_trusted_certificate
  input.proxy_ssl_verify == "on"
  msg := "proxy_ssl_trusted_certificate must be explicitly set when proxy_ssl_verify is on"
}

4. Synthetic Monitoring on the Handshake Path

Don't wait for production alerts. Add a synthetic check that measures TLS handshake time specifically:

# Prometheus blackbox exporter probe config
modules:
  https_upstream_tls:
    prober: tcp
    timeout: 5s
    tcp:
      tls: true
      tls_config:
        ca_file: /etc/ssl/private/internal-ca-chain.pem
      query_response:
        - expect: ""

Alert if handshake time exceeds 2 seconds — that's your early warning before the 110 timeout fires in Nginx.