RAPID INCIDENT TRIAGE & ACTION PLAN•Performance
01. Error Signature
PostgreSQL Long-Running Query SSL Expiry
02. Root Cause Vector

A long-running PostgreSQL query drops mid-execution because the SSL/TLS session times out or is terminated by an intermediate network dev...

03. Rapid Prescription

Enforce consistent lock order (e.g. ORDER BY id FOR UPDATE) or use non-blocking SKIP LOCKED.

Zero-Trust Client IDE/ Performance

PostgreSQL Long-Running Query SSL Expiry

A long-running PostgreSQL query drops mid-execution because the SSL/TLS session times out or is terminated by an intermediate network device, killing the connection before the query completes.

SYSTEM STATUS
CREDITS: 0 / 3
source.yaml(60L)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
# legacy-app/docker-compose.yml (DO NOT TOUCH - PCI ZONE)
# TODO: Migrate to Kubernetes before Q4 2025
# WARNING: SSL certs auto-renew via cron, but sometimes they don't sync...

version: '3.5'

services:
  postgres-primary:
    image: my-registry.internal/legacy/postgres:12.8-alpine  # Stuck on old version due to compliance
    container_name: pci-db-primary
    environment:
      - POSTGRES_DB=legacy_transactions
      - POSTGRES_USER=svc_pci_app
      # VULNERABILITY: SSL cert lifetime (90d) mismatches with connection timeout & long query kill time, causing mid-query expiry.
      - POSTGRES_SSL_CERT=/certs/server.crt
      - POSTGRES_SSL_KEY=/certs/server.key
      - AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
      - AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    volumes:
      - ./legacy_certs:/certs:ro  # Certs renewed by external cron, but container not restarted
      - ./init.sql:/docker-entrypoint-initdb.d/init.sql
    command: >
      postgres
      -c ssl=on
      -c ssl_cert_file=/certs/server.crt
      -c ssl_key_file=/certs/server.key
      -c ssl_ca_file=/certs/ca.crt
      -c statement_timeout=0  # Disabled for "critical batch jobs"
      -c idle_in_transaction_session_timeout=0
    networks:
      - pci-secure-zone

  batch-processor:
    image: my-registry.internal/chaos/batch:v2.1.3
    container_name: legacy-batch-runner
    environment:
      - DATABASE_URL=postgresql://svc_pci_app:Th1sIsAL0ngS3cretP@ssw0rd!@postgres-primary:5432/legacy_transactions?sslmode=verify-full&sslrootcert=/certs/ca.crt
      - STRIPE_API_KEY=sk_live_4eC39HqLyjWDarjtT1zdp7dc
      - PRIVATE_SSH_KEY=-----BEGIN OPENSSH PRIVATE KEY-----\nb3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABlwAAAAdzc2gtcn\nNhAAAAAwEAAQAAAYEAvYkHp8e6vG7K9pKjG8ZJzT8mW9Q6yN1LpM8K1pLmHqY7Kj8X7zT\n-----END OPENSSH PRIVATE KEY-----
    volumes:
      - ./scripts:/scripts
    command: >
      bash -c '
      # This long-running analytic query sometimes fails after cert refresh
      /scripts/run_legacy_aggregation.sh
      '
    networks:
      - pci-secure-zone
    depends_on:
      - postgres-primary

networks:
  pci-secure-zone:
    driver: bridge
    # internal: true  # Commented out for "debugging" last year

# init.sql below mounts but is ignored if DB already exists...
-- CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
-- -- TODO: Index this massive table
-- SELECT * FROM transactions WHERE created_at < NOW() - INTERVAL '365 days';
AI Audit Console
> Waiting for pre-flight audit...
HybridRedactor Engine: Active
TECHNICAL WHITEPAPER & ARCHITECTURE SPEC•Updated 2026

How to Fix PostgreSQL 'SSL Connection Closed Unexpectedly' on Long-Running Queries

Threat/Impact Level: HIGH | Downtime Risk: HIGH | Time to Fix: 15–30 mins


TL;DR

  • What broke: PostgreSQL dropped an active SSL session mid-query because a network appliance (ALB, NAT Gateway, firewall) or the server itself killed the idle-looking TCP connection before the query returned results.
  • How to fix it: Tune TCP keepalive settings at the OS and PostgreSQL level, increase or disable SSL renegotiation, and configure your load balancer idle timeout to exceed your longest expected query runtime.
  • Fast path: Use our Client-Side Sandbox above to auto-refactor your postgresql.conf or connection string — paste your config and get patched output instantly.

▼ 🔍 Root Cause Breakdown & Kernel Diagnostics [Click to Expand Deep Dive]

The Incident (What Does the Error Mean?)

FATAL:  SSL connection has been closed unexpectedly
server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.

This is not a query bug. The query was running fine. The TCP socket underneath the SSL session was silently torn down — by a firewall idle-timeout, an AWS ALB 60-second hard timeout, a NAT Gateway 350-second UDP/TCP timeout, or OpenSSL's built-in SSL renegotiation kicking in on long-lived connections. PostgreSQL sees the socket die and throws FATAL. Your transaction is rolled back. Any work done is lost.


▼ 🔍 Detailed Impact & Blast Radius Analysis [Click to Expand Deep Dive]

The Attack Vector / Blast Radius

This is a silent data integrity risk disguised as a network error.

  • Long ETL jobs, pg_dump over SSL, bulk INSERT ... SELECT — all silently aborted. If your application doesn't check for this specific error and retry with proper transaction handling, you get partial writes with no alert.
  • Connection poolers (PgBouncer, RDS Proxy) make this worse. The pooler may hold the dead connection in its pool and hand it to the next client, causing a cascade of SSL connection closed unexpectedly errors across unrelated application threads.
  • AWS RDS / Aurora specific: The RDS proxy and the underlying ALB both enforce a hard 60-second idle timeout by default. A query that runs for 90 seconds with no intermediate network traffic (no rows streamed back yet) looks "idle" to the load balancer. It tears down the TCP connection. PostgreSQL never knew it was coming.
  • On-prem / GCP / Azure: Stateful firewalls with TCP idle timeouts of 30–300 seconds are the usual culprit. The query is CPU-bound on the server, producing no network traffic, so the firewall ages out the flow entry and sends a RST.

How to Fix It

Basic Fix — TCP Keepalive at the OS Level

On the PostgreSQL server host, reduce the TCP keepalive probe interval so the kernel sends keepalive packets before any firewall or load balancer times out the connection.

# /etc/sysctl.conf
- # net.ipv4.tcp_keepalive_time = 7200
- # net.ipv4.tcp_keepalive_intvl = 75
- # net.ipv4.tcp_keepalive_probes = 9
+ net.ipv4.tcp_keepalive_time = 60
+ net.ipv4.tcp_keepalive_intvl = 10
+ net.ipv4.tcp_keepalive_probes = 6

Apply without reboot:

sysctl -p

This makes the kernel send a keepalive probe after 60 seconds of silence, every 10 seconds, for 6 attempts — totaling 120 seconds before the kernel declares the connection dead. This keeps the TCP flow alive through most firewalls.


Enterprise Best Practice — PostgreSQL + Application Layer

1. Enable keepalives in postgresql.conf:

# postgresql.conf
- #tcp_keepalives_idle = 0
- #tcp_keepalives_interval = 0
- #tcp_keepalives_count = 0
+ tcp_keepalives_idle = 60
+ tcp_keepalives_interval = 10
+ tcp_keepalives_count = 6

2. Disable SSL renegotiation (PostgreSQL < 14 only — this was the primary culprit in older versions):

# postgresql.conf (PostgreSQL 9.x–13.x)
- #ssl_renegotiation_limit = 512MB
+ ssl_renegotiation_limit = 0

PostgreSQL 14+ removed ssl_renegotiation_limit entirely. If you're still on 12 or 13, set it to 0 to disable forced renegotiation that can interrupt long transfers.

3. Pass keepalive parameters in your connection string (application side):

# Python psycopg2 / libpq connection string
- postgresql://user:pass@host:5432/db
+ postgresql://user:pass@host:5432/db?keepalives=1&keepalives_idle=60&keepalives_interval=10&keepalives_count=6
# JDBC (Java)
- jdbc:postgresql://host:5432/db
+ jdbc:postgresql://host:5432/db?tcpKeepAlive=true&socketTimeout=0

4. AWS RDS / Aurora — fix the load balancer timeout:

If you're behind an Application Load Balancer, the default idle timeout is 60 seconds. For long-running queries this is fatal.

# Terraform — aws_lb resource
 resource "aws_lb" "main" {
   name = "app-lb"
-  idle_timeout = 60
+  idle_timeout = 4000
 }

For RDS Proxy, set ConnectionBorrowTimeout and ensure your proxy is not using MaxConnectionsPercent so aggressively that connections are recycled mid-query.


💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.

(For CI/CD pipeline automation, install via npm i -g stackengine).


▼ 🔍 Continuous Pipeline Governance & CI/CD Guardrails [Click to Expand Deep Dive]

Prevention in CI/CD

1. Checkov — flag missing keepalive on RDS Proxy / ALB Terraform:

Write a custom Checkov check or use terraform-compliance to assert idle_timeout on aws_lb resources is never below your P99 query duration SLA.

# terraform-compliance feature file
Scenario: ALB idle timeout must support long-running queries
  Given I have aws_lb defined
  Then it must contain idle_timeout
  And its value must be greater than 300

2. pg_stat_activity alerting — catch long queries before they hit the timeout:

-- Alert if any query has been running > 5 minutes
SELECT pid, now() - query_start AS duration, query, state
FROM pg_stat_activity
WHERE state = 'active'
  AND now() - query_start > interval '5 minutes';

Pipe this into your Prometheus postgres_exporter custom query config and alert before the SSL timeout window closes.

3. statement_timeout as a safety net — never let runaway queries accumulate:

# postgresql.conf
- #statement_timeout = 0
+ statement_timeout = '30min'

Set this at the role level for application users, not globally, to avoid killing legitimate long-running maintenance jobs:

ALTER ROLE app_user SET statement_timeout = '30min';
ALTER ROLE etl_user SET statement_timeout = '4h';

4. Integration test — simulate the timeout in your pipeline:

Use tc (traffic control) in your CI environment to simulate a 60-second TCP idle drop and assert your application retries correctly with exponential backoff.

Official Grounding & Normative Specifications

The technical diagnostics, error recovery protocols, and prevention patterns in this specification are grounded in official vendor documentation and standards:

Diagnostic & Architectural FAQ

Q1.Why does the SSL connection close only on long-running queries and not short ones?
Short queries complete before any network device's idle timeout fires. Long-running queries that are CPU-bound on the server produce zero network traffic for extended periods. Stateful firewalls, AWS ALBs, and NAT Gateways interpret this silence as an abandoned connection and send a TCP RST, tearing down the SSL session. The query was healthy — the network killed it.
Q2.Does setting tcp_keepalives_idle in postgresql.conf override the OS sysctl settings?
Yes, per-connection. PostgreSQL uses the libpq setsockopt() calls to override the OS defaults on each individual socket it manages. However, setting both the OS sysctl values and the postgresql.conf parameters is the correct approach — the sysctl values cover any connections or tools that bypass libpq's socket options. Manually configuring this across your infrastructure is highly error-prone and risky; a single misconfiguration can introduce security vulnerabilities or cause widespread connection failures. Drop the broken manifest into the StackEngine Local Sandbox above to automatically compile a zero-trust fix.
Q3.We use AWS RDS — we can't edit postgresql.conf directly. How do we apply tcp_keepalives settings?
For RDS, set tcp_keepalives_idle, tcp_keepalives_interval, and tcp_keepalives_count in your RDS Parameter Group and apply it to your instance. For the load balancer timeout issue, increase the ALB idle_timeout to exceed your longest query runtime. For RDS Proxy, review the proxy's connection borrow timeout and ensure your VPC Security Groups and Network ACLs are not enforcing their own idle timeouts shorter than your query duration. Doing this across large mono-repos or at scale is a compliance nightmare. Deploy the StackEngine CI/CD Automator (Team Edition) for on-prem, automated pull request generation with Maker-Checker review.