Nginx gRPC HTTP/1.1 Fallback Premature Close

The Incident (What Does the Error Mean?)

Raw error from nginx/error.log:

2024/01/15 03:42:17 [error] 31#31: *8472 upstream prematurely closed connection
while reading response header from upstream, client: 10.0.1.45,
server: api.internal, request: "POST /helloworld.Greeter/SayHello HTTP/2.0",
upstream: "http://127.0.0.1:50051/helloworld.Greeter/SayHello",

What is actually happening: Nginx opened an HTTP/1.1 connection to the gRPC backend (port 50051). The gRPC server speaks HTTP/2 with binary framing. The moment the server sees an HTTP/1.1 POST instead of an HTTP/2 PRI * HTTP/2.0 preface, it resets the connection. Nginx reads a closed socket where it expected response headers and emits 502. Every request fails — not intermittently, but 100% of the time once the backend is exclusively gRPC.

For HTTP/1.1 fallback scenarios (transcoding via grpc-gateway or Envoy), the failure mode shifts: the fallback route uses proxy_pass correctly, but the primary gRPC route is missing grpc_pass, so all native gRPC clients (mobile, service mesh) are dead while REST clients survive. This creates a split-brain outage that is harder to triage.

The Attack Vector / Blast Radius

This is a full service outage vector, not a degradation. The blast radius:

• All gRPC clients return UNAVAILABLE (status 14) — most client libraries will retry with backoff, hammering the backend with connection attempts that all fail at the HTTP/2 handshake layer.
• Retry storms: gRPC client-side retries (via ServiceConfig or interceptors) will amplify traffic 3–5x against an already-broken upstream. If you have a retry budget of 3, your backend sees 3x the original RPS of pure connection setup overhead.
• Long-lived streams (server-streaming, bidirectional): Even if you partially fix the config, incorrect timeout values (grpc_read_timeout defaulting to 60s) will silently kill streaming RPCs that are legitimately idle between messages — e.g., a real-time feed waiting for the next event. The client gets EOF with no error code.
• Health check false positives: If your load balancer health check hits a REST endpoint (/healthz), it will report the backend as healthy while 100% of gRPC traffic is failing. You will not get paged.
• HTTP/1.1 fallback misconfiguration creates a security-adjacent issue: if proxy_pass is left in place alongside grpc_pass on the same location block, Nginx behavior is undefined across versions — some versions silently prefer proxy_pass, meaning your gRPC traffic is being attempted over HTTP/1.1 and response bodies from failed handshakes may be logged or forwarded to clients, leaking internal server error strings.

Prevention in CI/CD

1. Nginx Config Linting in Pre-Commit

# .pre-commit-config.yaml
- repo: https://github.com/nginxinc/nginx-lint
  hooks:
    - id: nginx-lint
      args: [--include, /etc/nginx/]

nginx -t does not catch semantic errors like proxy_pass on a gRPC route. You need a purpose-built linter or a test harness.

2. Integration Test with `grpcurl` in CI

Add this to your pipeline before any Nginx config deploy:

#!/bin/bash
# ci/test-grpc-proxy.sh
set -e

NGINX_HOST="${NGINX_TEST_HOST:-localhost:443}"

echo "[TEST] Verifying gRPC proxy is not returning 502..."
RESPONSE=$(grpcurl -plaintext -d '{"name": "ci-test"}' \
  "$NGINX_HOST" helloworld.Greeter/SayHello 2>&1)

if echo "$RESPONSE" | grep -q "upstream prematurely closed\|502\|UNAVAILABLE"; then
  echo "[FAIL] gRPC proxy misconfiguration detected."
  exit 1
fi

echo "[PASS] gRPC proxy healthy."

3. OPA/Conftest Policy for Nginx Configs

# policy/nginx_grpc.rego
package nginx

deny[msg] {
  input.servers[_].locations[loc]
  loc.path == "/"
  loc.proxy_pass
  not loc.grpc_pass
  contains(loc.proxy_pass, "50051")  # known gRPC port
  msg := sprintf(
    "Location '%v' routes to gRPC port 50051 using proxy_pass. Use grpc_pass instead.",
    [loc.path]
  )
}

Run in CI:

conftest test nginx.conf --policy policy/nginx_grpc.rego

4. Prometheus Alert on gRPC 502 Rate

# alerting/grpc_502.yml
groups:
  - name: nginx_grpc
    rules:
      - alert: NginxGrpcUpstreamPrematureClose
        expr: |
          rate(nginx_http_requests_total{
            status="502",
            request=~".*/grpc.*"
          }[2m]) > 0.1
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Nginx gRPC upstream returning 502 — check grpc_pass config"
          runbook_url: "https://your-wiki/runbooks/nginx-grpc-502"

This fires within 1 minute of the misconfiguration going live — before your on-call gets a Slack flood from application teams.