How to Fix Nginx 502 Bad Gateway: 'upstream sent too big header' Caused by Large Set-Cookie
Threat/Impact Level: HIGH | Downtime Risk: HIGH | Time to Fix: 5–10 mins
TL;DR
- What broke: Nginx's
proxy_buffer_size(default 4KB) is too small to hold the upstream's response headers — specifically an oversizedSet-Cookieheader (JWT payloads, serialized session blobs, or multi-domain cookie chains routinely exceed 4–8KB). - How to fix it: Increase
proxy_buffer_size,proxy_buffers, andproxy_busy_buffers_sizein the offendingserver {}orlocation {}block. - Fast path: Use our Client-Side Sandbox above to auto-refactor this — paste your
nginx.confand it generates the corrected buffer directives without sending your config to any external server.
▼ 🔍 Root Cause Breakdown & Kernel Diagnostics [Click to Expand Deep Dive]
The Incident (What Does the Error Mean?)
Your Nginx error log reads:
2024/01/15 03:42:17 [error] 1234#1234: *5678 upstream sent too big header while reading response header from upstream,
client: 203.0.113.44, server: api.example.com, request: "POST /auth/login HTTP/1.1",
upstream: "http://10.0.1.25:8080/auth/login", host: "api.example.com"The client receives:
HTTP/1.1 502 Bad Gateway
Server: nginx/1.24.0What Nginx is doing: When proxying, Nginx reads the upstream's response headers into a fixed-size buffer before forwarding them to the client. The moment the cumulative header size exceeds proxy_buffer_size (default: 4KB on most builds, occasionally 8KB), Nginx aborts the connection and logs this error. The upstream response — including your auth token, session cookie, or OIDC state — is silently dropped. The user sees a 502. Your on-call gets paged.
Common upstream culprits:
- JWT stored whole in
Set-Cookie(RS256 tokens are routinely 1–3KB each) - Multi-tenant apps setting 10–20 separate
Set-Cookielines - OIDC/OAuth2 proxies (Keycloak, Okta agent, oauth2-proxy) setting
_oauth2_proxycookies with full ID token payloads - Spring Session or Ruby on Rails encrypted cookie stores
▼ 🔍 Detailed Impact & Blast Radius Analysis [Click to Expand Deep Dive]
The Attack Vector / Blast Radius
This is not a security vulnerability in the traditional sense — but the blast radius in production is severe and the failure mode is deceptive:
1. Silent auth failures at scale. Login endpoints, OAuth2 callback URLs, and SSO assertion consumers are the most common trigger points. Every affected user gets a 502 on the exact request that was supposed to establish their session. Retry logic re-hits the same wall.
2. Monitoring blind spots. Your upstream application logged a 200 OK. Your APM shows the request succeeded. Only Nginx's error log tells the truth — and most teams don't have alerts on upstream sent too big header. The mismatch between upstream 200 and downstream 502 makes this extremely hard to triage without log correlation.
3. Cascading failure under autoscaling. If the oversized cookie is set on every response (e.g., a session refresh middleware that unconditionally rewrites the cookie), 100% of requests to that upstream fail. Load balancers mark the upstream unhealthy. Autoscaling spins new instances that are equally broken. You now have a full service outage, not a partial one.
4. Security regression risk from the wrong fix. The naive fix is to set proxy_buffer_size 64k globally in http {}. This allocates 64KB of kernel buffer memory per connection, per worker. On a busy proxy with 10,000 concurrent connections and 4 workers, that's 2.56GB of buffer memory — before you've served a single byte of response body. This is how you turn a 502 into an OOM kill.
How to Fix It (The Solution)
Root Cause Confirmation
Before touching config, confirm the actual header size from the upstream:
# Hit the upstream directly (bypassing Nginx) and measure response headers
curl -sI http://10.0.1.25:8080/auth/login -X POST \
-H 'Content-Type: application/json' \
-d '{"user":"test","pass":"test"}' \
| awk 'BEGIN{s=0} {s+=length($0)} END{print "Total header bytes:", s}'
# Or get the raw Set-Cookie size specifically
curl -sI http://10.0.1.25:8080/auth/login ... | grep -i set-cookie | wc -cIf the output is >4096 bytes, you've confirmed the root cause.
Basic Fix (Targeted Location Block)
Apply buffer overrides only to the location that generates large headers. Do not touch the global http {} block.
server {
listen 443 ssl;
server_name api.example.com;
location /auth/ {
proxy_pass http://upstream_backend;
-
+ # Increase header buffer to handle large Set-Cookie / JWT payloads
+ proxy_buffer_size 16k;
+ proxy_buffers 4 16k;
+ proxy_busy_buffers_size 32k;
}
location / {
proxy_pass http://upstream_backend;
# Default buffers are fine for non-auth endpoints
}
}Directive breakdown:
| Directive | Default | Recommended | Purpose |
|---|---|---|---|
proxy_buffer_size |
4k/8k | 16k |
Size of buffer for first part of response (headers) |
proxy_buffers |
8 4k/8k |
4 16k |
Number and size of buffers for response body |
proxy_busy_buffers_size |
8k/16k | 32k |
Max size of buffers being sent to client while reading |
Enterprise Best Practice (Multi-Upstream, Named Upstreams, Includes)
In environments with multiple upstreams or microservices where several endpoints may set large headers (OIDC callback, token introspection, profile endpoints), use a reusable include file and apply it selectively.
# /etc/nginx/snippets/large-header-buffers.conf (NEW FILE)
+proxy_buffer_size 16k;
+proxy_buffers 4 16k;
+proxy_busy_buffers_size 32k;
+proxy_max_temp_file_size 0;# /etc/nginx/sites-available/api.example.com
upstream auth_backend {
server 10.0.1.25:8080;
keepalive 32;
}
upstream app_backend {
server 10.0.1.30:8080;
keepalive 64;
}
server {
listen 443 ssl http2;
server_name api.example.com;
# Standard app traffic — default buffers
location / {
proxy_pass http://app_backend;
}
# Auth/SSO endpoints — oversized Set-Cookie headers expected
location ~ ^/(auth|oauth2|saml|sso)/ {
proxy_pass http://auth_backend;
+ include snippets/large-header-buffers.conf;
}
# Keycloak admin console also sets large headers
location /admin/ {
proxy_pass http://auth_backend;
+ include snippets/large-header-buffers.conf;
}
}After editing, always validate before reload:
nginx -t && nginx -s reload
# Confirm the fix — should return 200/302 instead of 502
curl -v https://api.example.com/auth/login -X POST ...
# Watch error log in real-time during smoke test
tail -f /var/log/nginx/error.log | grep 'too big header'💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.
(For CI/CD pipeline automation, install via
npm i -g stackengine).
▼ 🔍 Continuous Pipeline Governance & CI/CD Guardrails [Click to Expand Deep Dive]
Prevention in CI/CD
This class of error is entirely preventable. Here's how to catch it before it hits production:
1. Nginx Config Linting in CI (gixy)
gixy is a static analysis tool for Nginx configs. Add it to your pipeline:
# .github/workflows/nginx-lint.yml
- name: Lint Nginx config with gixy
run: |
pip install gixy
gixy /etc/nginx/nginx.conf
# gixy checks for proxy_buffer_size misconfigurations and other issues2. Integration Test: Assert Header Size
Add a contract test that fires against your staging upstream and asserts that response headers stay within bounds:
#!/bin/bash
# ci/check-header-size.sh
MAX_HEADER_BYTES=8192 # 8KB — safe under default 16k buffer
ACTUAL=$(curl -sI "$STAGING_AUTH_URL" ... | wc -c)
if [ "$ACTUAL" -gt "$MAX_HEADER_BYTES" ]; then
echo "FAIL: Response headers are ${ACTUAL} bytes, exceeding ${MAX_HEADER_BYTES} byte limit"
exit 1
fi
echo "PASS: Header size ${ACTUAL} bytes is within limit"3. OPA/Conftest Policy for Nginx (Rego)
If you manage Nginx config as code (Ansible, Helm, Terraform), enforce buffer minimums with a Conftest policy:
# policy/nginx_buffers.rego
package nginx
deny[msg] {
location := input.servers[_].locations[_]
location.proxy_pass # only check proxied locations
not location.proxy_buffer_size
msg := sprintf(
"Location '%v' is missing proxy_buffer_size. Auth endpoints require >= 16k.",
[location.path]
)
}4. Upstream Cookie Hygiene (Fix the Source)
The buffer increase is the right Nginx fix, but also address the upstream:
- JWT in cookies: Store only the JWT reference (session ID) in the cookie. Keep the token server-side in Redis/Memcached.
- Audit cookie payloads:
Set-Cookieheaders should rarely exceed 4KB. If they do, you have an architectural smell. - SameSite + Secure + HttpOnly: While you're touching cookie config, ensure these attributes are set. Large cookies that are also insecure are doubly problematic.
# Example: oauth2-proxy config reduction
- cookie_secret: "<full-jwt-payload-stored-in-cookie>"
+ session_store_type: "redis"
+ redis_connection_url: "redis://session-store:6379"
# Cookie now contains only a short opaque session ID, not the full tokenOfficial Grounding & Normative Specifications
The technical diagnostics, error recovery protocols, and prevention patterns in this specification are grounded in official vendor documentation and standards:
Official runtime configuration documentation governing lock acquisition timeouts, deadlock detection cycles, and wait logging.
Federal security guidelines for secret exfiltration prevention, automated integrity checks, and peer sign-offs.
Core memory subsystem controls, pressure stall information (PSI), and kernel OOM invocation mechanics.
Industry-standard specification for container execution boundaries and POSIX namespace isolation.