How to Fix Nginx auth_request 403 Forbidden: Debugging Subrequest Auth Failures in Production

Threat/Impact Level: CRITICAL | Exploitability/Downtime Risk: HIGH | Time to Fix: 15 mins

TL;DR

• What broke: Nginx's auth_request directive fired a subrequest to your auth endpoint, which returned 403 Forbidden. Nginx treats any non-2xx response as a hard block — every downstream request is now dead.
• How to fix it: The auth service is either rejecting the forwarded headers (missing Authorization, wrong Host, stripped cookies), the auth endpoint URL is wrong, or the auth service itself has a bug. Fix the header-forwarding config and validate the auth endpoint returns 200 for valid tokens.
• CTA: Use our Client-Side Sandbox above to paste your nginx.conf and auto-diagnose the misconfigured auth_request block without leaking your internal URLs or tokens.

The Incident (What does the error mean?)

You see this in your Nginx error log:

2024/01/15 03:42:17 [warn] 28#28: *1 auth request unexpected status: 403 while sending to client
2024/01/15 03:42:17 [error] 28#28: *1 auth_request forbidden: subrequest returned 403, client: 10.0.1.45, server: api.internal, request: "GET /protected/resource HTTP/1.1"

Nginx's auth_request module works as a synchronous gate. It fires an internal subrequest to the URI defined in auth_request /auth;. If that subrequest returns:

• 2xx → request proceeds to the upstream
• 401 → Nginx returns 401 to the client
• 403 → Nginx returns 403 to the client and logs this exact error
• Any other non-2xx → Nginx returns 500

A 403 specifically means your auth service received the request, understood it, and explicitly denied it. This is not a network timeout or a misconfigured proxy_pass URL — the auth service is reachable and is making an active access decision. Every single request hitting this location block is now returning 403 to end users.

The Attack Vector / Blast Radius

This failure mode is deceptively dangerous for two opposing reasons:

Scenario A — Legitimate users are locked out (Availability impact): The auth service is denying valid tokens because Nginx is stripping the Authorization header, Cookie, or X-Forwarded-* headers before the subrequest fires. The auth service sees an unauthenticated request and correctly returns 403. Your entire protected surface is down. SLA breach in progress.

Scenario B — The auth service logic is wrong (Security impact): If you "fix" this by switching your auth service to return 200 for all requests to unblock the outage, you've just disabled authentication entirely. This is the most common panic-driven mistake during an incident. Any unauthenticated actor can now reach your upstream services directly.

Blast radius: Every location block using this auth_request directive is affected simultaneously. In a microservices gateway pattern, this typically means all internal services behind the Nginx reverse proxy are either fully inaccessible or (if someone panic-patched the auth service) fully exposed.

Additionally: if proxy_pass_request_headers is off or proxy_set_header is overriding the Authorization header to an empty string, the auth service will log thousands of failed auth attempts — burning through rate limits and polluting your SIEM with false-positive alerts.

How to Fix It (The Solution)

Root Cause Checklist (run through these in order)

1. Headers not forwarded to auth subrequest — auth_request subrequests do NOT automatically inherit the parent request's headers in all configurations.
2. Wrong auth endpoint URI — The auth_request URI is an internal Nginx location, not a direct proxy URL.
3. Auth service is checking Host header — Nginx rewrites Host on subrequests.
4. Cookie-based auth with proxy_pass — Cookies are stripped unless explicitly forwarded.
5. The auth service has a bug — It's returning 403 for a valid token due to a deployment issue.

Basic Fix — Forward the Authorization header correctly

The most common cause: the Authorization header is silently dropped on the subrequest.

server {
    listen 443 ssl;
    server_name api.internal;

    location /protected/ {
        auth_request /auth;
+       auth_request_set $auth_status $upstream_status;

        proxy_pass http://backend_upstream;
    }

    location = /auth {
        internal;
        proxy_pass http://auth-service:8080/validate;

-       # Missing: headers not forwarded, auth service sees anonymous request
+       proxy_pass_request_body off;
+       proxy_set_header Content-Length "";
+       proxy_set_header X-Original-URI $request_uri;
+       proxy_set_header X-Original-Method $request_method;
+       proxy_set_header Authorization $http_authorization;
+       proxy_set_header Cookie $http_cookie;
+       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

Why proxy_pass_request_body off + Content-Length "": Auth validation endpoints only need headers. Forwarding the body to the auth service on every request is wasteful and can cause issues with streaming or large payloads.

Enterprise Best Practice — Full hardened auth_request pattern with error interception

server {
    listen 443 ssl;
    server_name api.internal;

+   # Capture auth failure codes and map them to proper client responses
+   error_page 401 = @error401;
+   error_page 403 = @error403;

    location /protected/ {
        auth_request /auth;
        auth_request_set $auth_status $upstream_status;
        auth_request_set $auth_user $upstream_http_x_auth_user;
        auth_request_set $auth_roles $upstream_http_x_auth_roles;

-       proxy_pass http://backend_upstream;
+       # Propagate auth-derived identity headers to the upstream service
+       proxy_set_header X-Auth-User $auth_user;
+       proxy_set_header X-Auth-Roles $auth_roles;
+       proxy_set_header X-Request-ID $request_id;
+       proxy_pass http://backend_upstream;
    }

    location = /auth {
        internal;

-       proxy_pass http://auth-service/validate;
+       proxy_pass http://auth-service:8080/v1/validate;
+       proxy_pass_request_body off;
+       proxy_set_header Content-Length "";
+       proxy_set_header Host $host;
+       proxy_set_header X-Original-URI $request_uri;
+       proxy_set_header X-Original-Method $request_method;
+       proxy_set_header X-Real-IP $remote_addr;
+       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+       proxy_set_header X-Forwarded-Proto $scheme;
+       proxy_set_header Authorization $http_authorization;
+       proxy_set_header Cookie $http_cookie;
+
+       # Timeouts: auth subrequest must be fast. If auth is slow, fix auth — not this timeout.
+       proxy_connect_timeout 5s;
+       proxy_read_timeout 10s;
+
+       # Cache 200 responses for 10s to reduce auth service load under high traffic
+       proxy_cache auth_cache;
+       proxy_cache_valid 200 10s;
+       proxy_cache_key "$http_authorization$cookie_session";
    }

+   location @error401 {
+       add_header WWW-Authenticate 'Bearer realm="api.internal"' always;
+       return 401 '{"error":"authentication_required"}';
+   }

+   location @error403 {
+       return 403 '{"error":"insufficient_permissions"}';
+   }
}

+# Auth response cache zone — define in http{} block
+proxy_cache_path /var/cache/nginx/auth levels=1:2 keys_zone=auth_cache:10m max_size=128m inactive=60s;

💡 Tired of pasting proprietary configs into ChatGPT? Generic AI tools log your company's ARNs, DB strings, and private keys. StackEngine is a zero-backend, pure Client-Side WASM utility. Drop your failing config into the sandbox above. We redact your secrets locally in the browser and auto-generate the refactored code using your own API key.

Prevention in CI/CD

1. Automated Nginx config linting in your pipeline

Never merge an Nginx config that hasn't been syntax-checked AND semantically validated:

# In your CI pipeline (GitHub Actions / GitLab CI)
nginx -t -c /path/to/nginx.conf

# Use gixy for security-focused static analysis of Nginx configs
pip install gixy
gixy /etc/nginx/nginx.conf
# gixy detects: missing auth headers, SSRF risks in proxy_pass, open redirects

2. Integration test the auth subrequest path explicitly

# In your staging environment smoke test suite
# Test 1: Valid token must return 200 from the protected route
curl -H "Authorization: Bearer $VALID_TOKEN" https://api.staging.internal/protected/health \
  --fail --silent --output /dev/null --write-out "%{http_code}" | grep -q 200

# Test 2: Missing token must return 401 (not 403, not 500)
curl https://api.staging.internal/protected/health \
  --fail-with-body --write-out "%{http_code}" | grep -q 401

# Test 3: Invalid token must return 401
curl -H "Authorization: Bearer invalidtoken" https://api.staging.internal/protected/health \
  --write-out "%{http_code}" | grep -q 401

3. OPA policy to enforce auth_request on all location blocks serving internal upstreams

# opa/policies/nginx_auth_required.rego
package nginx.security

deny[msg] {
    block := input.nginx.servers[_].locations[_]
    contains(block.proxy_pass, "backend_upstream")
    not block.auth_request
    msg := sprintf("Location block proxying to upstream '%v' is missing auth_request directive", [block.proxy_pass])
}

4. Monitor auth subrequest error rates in production

Add this to your Nginx log format and alert on it in Datadog/Prometheus:

log_format auth_debug '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" auth_status=$auth_status';

Alert threshold: auth_status=403 rate > 1% of total requests over 5 minutes → PagerDuty. A sudden spike in 403s from the auth subrequest is either a deployment breaking your auth service or a credential-stuffing attack hitting your rate limiter.