PostgreSQL pg_wal Disk Space Panic Monitor

The Incident (What does the error mean?)

Raw error from PostgreSQL logs:

PANIC: could not write to file "pg_wal/xlogtemp.7F4A2B": No space left on device
LOG:  startup process (PID 1234) was terminated by signal 6: Aborted
LOG:  aborting startup due to startup process failure

PostgreSQL's WAL writer attempted to flush a WAL segment to $PGDATA/pg_wal/ and received ENOSPC from the kernel. This is not a graceful error — PostgreSQL issues a PANIC, which means the postmaster process aborts immediately. There is no partial recovery. Every in-flight transaction is rolled back. Your application is throwing connection refused errors right now.

The pg_wal/ directory is on a dedicated partition (or shares one with $PGDATA). Either way, it is at 100% utilization.

The Attack Vector / Blast Radius

This is a cascading storage failure, not a single-point event. The usual kill chain:

1. Stale replication slot — A physical or logical replication slot exists for a replica that is lagging or dead. PostgreSQL cannot remove WAL segments that the slot's restart_lsn hasn't consumed. Segments accumulate indefinitely.
2. Broken archive_command — If archive_mode = on and your archive_command (S3, NFS, whatever) starts failing silently, PostgreSQL retains every unarchived segment. pg_wal/ grows until disk death.
3. wal_keep_size set too aggressively — A well-intentioned but oversized value forces retention of gigabytes of segments regardless of replication state.
4. Blast radius: Once PANIC fires — all databases on the cluster are offline, connection poolers (PgBouncer, RDS Proxy) start queuing or dropping connections, downstream services begin throwing 500s, and any replica in streaming replication enters a reconnect loop. If this is a primary in a HA pair, your failover automation may promote a replica that is itself behind — meaning potential data divergence.

Check this first — stale slots are the #1 cause:

SELECT slot_name, slot_type, active, restart_lsn,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal
FROM pg_replication_slots
ORDER BY retained_wal DESC;

If retained_wal is in the tens or hundreds of GB and active = false — that slot is your culprit.

Prevention in CI/CD

1. Alerting — Non-Negotiable Thresholds

# Prometheus alerting rule (alerts.yml)
groups:
  - name: postgres_wal
    rules:
      - alert: PgWalDiskUsageCritical
        expr: (
          node_filesystem_size_bytes{mountpoint="/var/lib/postgresql"} -
          node_filesystem_free_bytes{mountpoint="/var/lib/postgresql"}
        ) / node_filesystem_size_bytes{mountpoint="/var/lib/postgresql"} > 0.80
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "pg_wal partition >80% full — replication slot audit required"

      - alert: PgStaleReplicationSlot
        expr: pg_replication_slots_pg_wal_lsn_diff_bytes{active="false"} > 5368709120  # 5GB
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Inactive replication slot retaining >5GB WAL"

2. Automated Slot Hygiene (Cron / Kubernetes CronJob)

#!/bin/bash
# Drop replication slots inactive for >2 hours with >2GB retained WAL
psql -U postgres -c "
  SELECT pg_drop_replication_slot(slot_name)
  FROM pg_replication_slots
  WHERE active = false
    AND pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) > 2147483648
    AND slot_name NOT IN (SELECT slot_name FROM pg_stat_replication);
"

3. Terraform / IaC — Enforce Disk Separation

# RDS / self-managed: always isolate pg_wal on its own EBS volume
- # pg_wal co-located with $PGDATA on single 100GB gp3 volume
+ resource "aws_ebs_volume" "pg_wal" {
+   availability_zone = var.az
+   size              = 200
+   type              = "gp3"
+   iops              = 6000
+   tags = { Name = "postgres-pg-wal-dedicated" }
+ }

4. Checkov Policy (IaC Scan)

Add a custom Checkov check asserting that any PostgreSQL instance definition includes a max_slot_wal_keep_size parameter — block the Terraform plan in CI if it's absent or set to -1.