Nginx gRPC Passthrough Connection Reset Fix

The Incident (What Does the Error Mean?)

Raw error log:

2024/01/15 03:42:17 [error] 31#31: *18423 recv() failed (104: Connection reset by peer)
while reading response header from upstream,
client: 10.0.1.45, server: api.internal,
request: "POST /helloworld.Greeter/SayHello HTTP/2.0",
upstream: "grpc://10.96.0.42:50051",
host: "api.internal"

What this means operationally:

Error 104 (ECONNRESET) is the kernel telling you the remote end sent a TCP RST packet. In a gRPC passthrough context, this is almost never a transient blip — it signals a structural misconfiguration between Nginx's connection lifecycle and your gRPC server's connection lifecycle.

Nginx established (or reused) a TCP connection to the upstream, sent the HTTP/2 HEADERS frame for the gRPC call, and then the upstream closed the connection before sending back a single byte of response headers. Nginx received the RST, logged 104, and returned a 502 to the client.

Immediate consequence: Every affected gRPC call returns HTTP 502. In high-traffic environments, this triggers retry storms. If your client uses exponential backoff without jitter, you now have a thundering herd amplifying the outage.

The Attack Vector / Blast Radius

This is a cascading availability failure, not a security exploit — but its blast radius is severe:

Root cause chain (most common):

1. Keepalive connection reuse on a dead socket. Nginx's keepalive directive pools connections to the upstream. If the upstream gRPC server (e.g., gRPC-Go, gRPC-Java) has a MaxConnectionAge of 30s but Nginx's keepalive_timeout is 75s, Nginx will attempt to reuse a connection the server already closed. The server sends RST. Nginx logs 104.

2. Missing http2 upstream directive. gRPC requires HTTP/2. If http2 is not declared on the upstream or the grpc_pass target isn't using the grpc:// scheme, Nginx negotiates HTTP/1.1 with the upstream. The gRPC server, expecting HTTP/2 frames, immediately resets the connection.

3. Kubernetes pod restarts / rolling deployments. A pod terminates mid-connection. The kernel on the pod sends RST. Nginx has no health-check configured and keeps routing to the dead endpoint from its keepalive pool.

4. Upstream gRPC server enforcing MaxConnectionIdle. gRPC servers actively close idle connections. If Nginx holds a pooled connection past the server's idle timeout, the next request on that connection gets RST.

Blast radius:

• All gRPC clients behind this Nginx receive 502s simultaneously when the pool of stale connections is exhausted.
• Retry logic amplifies upstream load 3–10x.
• Health checks on the upstream may pass (TCP handshake succeeds to a new connection) while application-level requests fail on reused connections — making this invisible to naive monitoring.

Prevention in CI/CD

Don't let this regress. Enforce these checks in your pipeline:

1. Nginx Config Linting with `nginx -t` in CI

# .github/workflows/nginx-lint.yml
- name: Validate Nginx Config
  run: |
    docker run --rm -v $(pwd)/nginx:/etc/nginx:ro nginx:latest \
      nginx -t -c /etc/nginx/nginx.conf

2. Conftest / OPA Policy — Enforce `grpc_pass` and `keepalive`

# policy/nginx_grpc.rego
package nginx.grpc

deny[msg] {
    input.upstreams[_].keepalive_timeout == ""
    msg := "UPSTREAM BLOCK: keepalive_timeout must be explicitly set for gRPC upstreams. Stale connections cause error 104."
}

deny[msg] {
    loc := input.servers[_].locations[_]
    contains(loc.path, "grpc") 
    not startswith(loc.grpc_pass, "grpc")
    msg := sprintf("LOCATION '%v': gRPC paths must use grpc_pass with grpc:// or grpcs:// scheme, not proxy_pass.", [loc.path])
}

3. Integration Test — Validate gRPC Passthrough End-to-End

# Use grpcurl in your smoke test suite
grpcurl -plaintext \
  -d '{"name": "ci-probe"}' \
  -max-time 5 \
  api.internal:443 \
  helloworld.Greeter/SayHello

# Assert exit code 0 and no 'Code: Unavailable' in output
if echo "$output" | grep -q 'Code: Unavailable'; then
  echo "FAIL: gRPC upstream returning UNAVAILABLE — check Nginx keepalive config"
  exit 1
fi

4. Alert on 104 Errors in Production

# Prometheus alerting rule
- alert: NginxGrpcConnectionReset
  expr: |
    increase(nginx_ingress_controller_requests_total{
      status="502",
      upstream_addr=~"grpc.*"
    }[5m]) > 10
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Nginx gRPC upstream returning 502 — likely recv() 104 connection reset"
    runbook: "https://wiki.internal/runbooks/nginx-grpc-104"

The definitive checklist before deploying any Nginx gRPC config:

• grpc_pass uses grpc:// or grpcs:// scheme (NOT proxy_pass)
• keepalive_timeout set below upstream gRPC server MaxConnectionAge
• keepalive connection pool size tuned to expected concurrency
• grpc_next_upstream error timeout configured for automatic retry on stale connections
• grpc_read_timeout matches longest expected streaming RPC duration
• Active health checks configured (Nginx Plus) or grpc_next_upstream_tries > 1 as fallback
• grpc_intercept_errors on with proper gRPC status code error pages