PostgreSQL Max WAL Senders Limit Fix

The Incident (What Does the Error Mean?)

Raw error output:

FATAL:  number of requested standby connections exceeds max_wal_senders
        (currently 10)

PostgreSQL maintains a fixed pool of WAL sender processes defined at startup. Every streaming replica, pg_basebackup job, logical replication publisher connection, and replication slot consumer eats one slot. When that pool is full, the next connection attempt dies immediately with this FATAL. The new replica does not connect. It does not retry silently. It fails hard.

If this is a primary→replica HA pair and the replica reconnects after a network blip, it will be refused — replica lag grows unbounded, and your failover target is now stale.

The Attack Vector / Blast Radius

This is a resource exhaustion / misconfiguration failure, not a security exploit, but the blast radius is severe:

1. Replica starvation. Any standby that can't connect stops applying WAL. Under high write load, pg_wal on the primary grows until wal_keep_size is exceeded or a replication slot holds WAL indefinitely — disk exhaustion on primary is now on the table.
2. Replication slot bloat. If you're using logical replication slots and the subscriber can't connect, the slot keeps accumulating WAL segments. This is one of the most common causes of unplanned primary disk-full outages.
3. Backup pipeline failure. pg_basebackup and pgBackRest streaming backups also consume WAL sender slots. A full max_wal_senders means your backup job silently fails — you discover this during a recovery drill, not before.
4. Cascading HA failure. Patroni, repmgr, and Stolon all use streaming replication health as a fencing signal. A replica that can't connect may be incorrectly fenced or promoted, causing split-brain.

Prevention in CI/CD

1. Checkov Policy — Catch Undersized WAL Senders in IaC

# checkov custom check: check_pg_wal_senders.py
from checkov.common.models.enums import CheckResult
from checkov.terraform.checks.resource.base_resource_check import BaseResourceCheck

class PgMaxWalSendersCheck(BaseResourceCheck):
    def __init__(self):
        super().__init__(
            name="Ensure max_wal_senders >= 20 for production RDS PG",
            id="CKV_CUSTOM_PG_001",
            supported_resources=["aws_db_parameter_group"]
        )

    def scan_resource_conf(self, conf):
        params = conf.get("parameter", [])
        for p in params:
            if p.get("name") == "max_wal_senders":
                if int(p.get("value", 0)) >= 20:
                    return CheckResult.PASSED
        return CheckResult.FAILED

2. Prometheus Alert — Catch Saturation Before It Bites

# prometheus/alerts/postgres.yml
groups:
  - name: postgres_replication
    rules:
      - alert: WalSendersSaturation
        expr: |
          pg_stat_replication_count / on(instance)
          pg_settings_max_wal_senders > 0.8
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "WAL senders at {{ $value | humanizePercentage }} capacity on {{ $labels.instance }}"
          runbook: "https://wiki.internal/runbooks/postgres-wal-senders"

3. Pre-deploy Smoke Test

#!/usr/bin/env bash
# ci/scripts/check_pg_capacity.sh
MAX=$(psql -At -c "SHOW max_wal_senders;")
ACTIVE=$(psql -At -c "SELECT count(*) FROM pg_stat_replication;")
SLOTS=$(psql -At -c "SELECT count(*) FROM pg_replication_slots WHERE active = false;")

if (( ACTIVE + SLOTS >= MAX - 3 )); then
  echo "FAIL: WAL sender headroom critically low ($ACTIVE active, $SLOTS idle slots, max=$MAX)"
  exit 1
fi
echo "OK: WAL sender capacity nominal ($ACTIVE/$MAX used)"

Plug this into your GitHub Actions pre-deploy job or Spinnaker gate. It catches saturation before a new replica or backup agent is provisioned.