STACKENGINE
CLI
Install Community CLI
RAPID INCIDENT TRIAGE & ACTION PLAN•Performance
Read Architecture SpecOpen Local Sandbox
01. Error Signature
Kubernetes OOMKilled Exit 137 Analyzer
02. Root Cause Vector

EKS Fargate pod crashes with OOMKilled (exit 137) due to misconfigured memory limits, triggering CrashLoopBackOff and cascading service d...

03. Rapid Prescription

Increase container memory limits or optimize runtime heap/cgroup footprint.

Zero-Trust Client IDE/ Performance

Kubernetes OOMKilled Exit 137 Analyzer

EKS Fargate pod crashes with OOMKilled (exit 137) due to misconfigured memory limits, triggering CrashLoopBackOff and cascading service degradation.

SYSTEM STATUS
CREDITS: 0 / 3
source.yaml(71L)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
apiVersion: v1
kind: Pod
metadata:
  name: legacy-payment-processor
  namespace: pci-secure-zone
  annotations:
    # TODO: This annotation is from the old on-prem migration, might conflict with Fargate scheduler
    scheduler.alpha.kubernetes.io/critical-pod: "true"
    # VULNERABILITY: Memory limit set lower than the JVM heap size, causing OOMKilled on startup under Fargate's memory accounting
    # The JVM inside the container will allocate -Xmx based on container memory, but the limit here is set too low.
spec:
  # Using Fargate spot for cost savings, but spot instances can have memory pressure variability
  nodeSelector:
    eks.amazonaws.com/compute-type: fargate
    eks.amazonaws.com/fargate-profile: legacy-app-profile
  containers:
  - name: payment-api
    image: my-registry.internal/legacy-java-app:stable-v2.1  # Old image, hasn't been updated in 2 years
    # Workaround for the old JVM not respecting cgroups memory limits, trying to set heap manually
    command: ["/bin/sh", "-c"]
    args:
      - |
        # TODO: Fix this memory calculation; currently sets heap to 90% of limit, but limit might be wrong
        export JAVA_OPTS="-Xmx$(($(cat /sys/fs/cgroup/memory/memory.limit_in_bytes) * 90 / 100 / 1024 / 1024))m"
        # Embedded dummy secret: AWS credentials for the legacy S3 backup (should be in Secrets Manager)
        export AWS_ACCESS_KEY_ID="AKIAIOSFODNN7EXAMPLE"
        export AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
        # Embedded dummy secret: Stripe API key for payment processing
        export STRIPE_API_KEY="sk_live_4eC39HqLyjWDarjtT1zdp7dc"
        # Start the app with the calculated heap
        exec java $JAVA_OPTS -jar /app/payment-service.jar
    resources:
      requests:
        memory: "256Mi"
        cpu: "250m"
      limits:
        # This limit is set too low for the JVM's -Xmx calculation and the app's baseline memory.
        # The JVM will try to allocate ~90% of 512Mi (~460Mi) for heap, but the container only has 512Mi total,
        # leaving no room for off-heap memory, metaspace, and OS processes, causing OOMKilled exit 137.
        memory: "512Mi"
        cpu: "500m"
    env:
    - name: DB_URI
      # Embedded dummy secret: PostgreSQL connection string with credentials
      value: "postgresql://payment_user:SuperSecretPassword123!@legacy-db.internal:5432/payment_db"
    - name: LOG_LEVEL
      value: "DEBUG"  # DEBUG logging increases memory usage slightly
    # Old liveness probe that might cause restarts under load, contributing to CrashLoopBackOff
    livenessProbe:
      httpGet:
        path: /health
        port: 8080
      initialDelaySeconds: 30
      periodSeconds: 10
      failureThreshold: 3
    # Redundant security context from an old compliance template, might not apply to Fargate
    securityContext:
      runAsUser: 1000
      runAsNonRoot: true
  # Commented-out old affinity rules from the node-based deployment
  # affinity:
  #   nodeAffinity:
  #     requiredDuringSchedulingIgnoredDuringExecution:
  #       nodeSelectorTerms:
  #       - matchExpressions:
  #         - key: kubernetes.io/arch
  #           operator: In
  #           values:
  #           - amd64
  restartPolicy: Always  # Default, but combined with OOMKilled leads to CrashLoopBackOff
AI Audit Console
> Waiting for pre-flight audit...
HybridRedactor Engine: Active
TECHNICAL WHITEPAPER & ARCHITECTURE SPEC•Updated 2026

Fixing Kubernetes OOMKilled Exit Code 137 CrashLoopBackOff on EKS Fargate Spot Instances

Threat/Impact Level: HIGH | Downtime Risk: HIGH | Time to Fix: 15–30 mins

TL;DR

  • What broke: Your container exceeded its resources.limits.memory ceiling. The Linux OOM killer sent SIGKILL (signal 9), producing exit code 137. Fargate's per-pod VM boundary made memory overcommit impossible, so the kill was immediate.
  • How to fix it: Raise resources.limits.memory to match actual working-set usage (get this from kubectl top pod), align requests to avoid Fargate vCPU/memory ratio mismatches, and add a VPA in recommendation mode.
  • Shortcut: Use our Client-Side Sandbox above to auto-refactor your pod spec — paste your YAML and get corrected limits without sending your config to a third-party server.
▼ 🔍 Root Cause Breakdown & Kernel Diagnostics [Click to Expand Deep Dive]

The Incident (What Does the Error Mean?)

Raw output from a live cluster:

$ kubectl get pods -n production
NAME                          READY   STATUS             RESTARTS   AGE
api-deployment-7d9f8b-xk2pq   0/1     CrashLoopBackOff   8          18m

$ kubectl describe pod api-deployment-7d9f8b-xk2pq -n production
...
Last State:  Terminated
  Reason:    OOMKilled
  Exit Code: 137
  Started:   Mon, 10 Jun 2024 03:12:44 +0000
  Finished:  Mon, 10 Jun 2024 03:12:51 +0000
Restart Count: 8
...
Limits:
  memory: 256Mi
Requests:
  memory: 128Mi

Exit code 137 = 128 + 9. Signal 9 is SIGKILL, issued by the kernel OOM killer the moment the container's RSS exceeded the cgroup memory limit. On EKS Fargate, each pod runs in an isolated microVM. There is no node-level memory to borrow. The moment your container's working set crosses limits.memory, the kill is unconditional and instantaneous — no grace period, no swap.

CrashLoopBackOff is Kubernetes applying exponential back-off (10s → 20s → 40s → … → 5min cap) because the container keeps dying on startup or under load. After 8 restarts, your back-off window is near maximum. The service is effectively down.

▼ 🔍 Detailed Impact & Blast Radius Analysis [Click to Expand Deep Dive]

The Attack Vector / Blast Radius

This is not just a single pod dying. Trace the cascade:

  1. Pod restart storm: Each restart re-pulls config, re-establishes DB connections, and re-warms caches. Under load, the pod dies before warm-up completes, causing the next restart to OOMKill even faster.
  2. Fargate Spot eviction amplifier: Spot capacity interruptions can coincide with OOM events. If your Fargate Spot profile has no on-demand fallback, the replacement pod may never schedule.
  3. HPA death spiral: If a Horizontal Pod Autoscaler is attached, it sees low READY pod count and scales up. New pods also OOMKill. You burn Fargate vCPU/memory allocation costs at scale while serving zero traffic.
  4. Fargate memory-to-vCPU ratio constraint: Fargate enforces specific vCPU/memory combinations. A limits.memory: 256Mi with requests.cpu: 1 is an invalid combination — Fargate silently rounds up memory, but your limit stays at 256Mi, so the container gets killed inside a larger-than-requested VM. This is the most common hidden cause of OOMKill on Fargate that engineers miss.
  5. Downstream dependency timeouts: Any service calling this pod accumulates open connections during CrashLoopBackOff. If your upstream has a short connect timeout, you get cascading 503s across your mesh.

The blast radius on a production API with 8 restarts already logged: full service outage for all traffic hitting this deployment.

How to Fix It

Step 1: Get Actual Memory Usage First

# Requires metrics-server or CloudWatch Container Insights enabled
kubectl top pod -n production --sort-by=memory

# For historical peak usage (CloudWatch Insights query):
fields @timestamp, pod_name, pod_memory_working_set_bytes
| filter ClusterName = 'your-cluster'
| stats max(pod_memory_working_set_bytes) by pod_name

Set limits.memory to at least 1.5× your observed peak working set. Never guess.

Basic Fix — Corrected Resource Block

 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: api-deployment
 spec:
   template:
     spec:
       containers:
       - name: api
         image: myrepo/api:v2.1.0
         resources:
           requests:
-            memory: "128Mi"
-            cpu: "250m"
+            memory: "512Mi"
+            cpu: "500m"
           limits:
-            memory: "256Mi"
-            cpu: "500m"
+            memory: "1Gi"
+            cpu: "1000m"

Why requests matter on Fargate: Fargate allocates the microVM based on requests, not limits. If requests.memory is 128Mi, Fargate provisions a 0.25 vCPU / 0.5GB VM tier. Your limit of 256Mi fits, but leaves zero headroom for JVM heap expansion, glibc malloc arenas, or Node.js V8 heap growth under load.

Enterprise Best Practice — VPA + LimitRange + Fargate Profile Alignment

+apiVersion: autoscaling.k8s.io/v1
+kind: VerticalPodAutoscaler
+metadata:
+  name: api-deployment-vpa
+  namespace: production
+spec:
+  targetRef:
+    apiVersion: apps/v1
+    kind: Deployment
+    name: api-deployment
+  updatePolicy:
+    updateMode: "Off"   # Recommendation mode only — apply manually first
+  resourcePolicy:
+    containerPolicies:
+    - containerName: api
+      minAllowed:
+        memory: 256Mi
+      maxAllowed:
+        memory: 4Gi
+      controlledResources: ["memory"]
---
+apiVersion: v1
+kind: LimitRange
+metadata:
+  name: fargate-memory-guardrails
+  namespace: production
+spec:
+  limits:
+  - type: Container
+    default:
+      memory: 1Gi
+    defaultRequest:
+      memory: 512Mi
+    max:
+      memory: 4Gi
+    min:
+      memory: 256Mi

Fargate profile — ensure your memory value is a valid Fargate tier:

 # eksctl cluster config or Terraform aws_eks_fargate_profile
 # Valid Fargate memory values: 512MB, 1GB, 2GB, 3GB, 4GB ... 30GB (in 1GB increments above 512MB)
-# requests.memory: 128Mi  ← rounds to 512MB VM, limit of 256Mi causes OOMKill in a 512MB VM
+# requests.memory: 512Mi  ← explicit 512MB tier, limit of 1Gi safely within tier

Check valid combinations: 0.25 vCPU → 0.5–2GB, 0.5 vCPU → 1–4GB, 1 vCPU → 2–8GB.

💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.

(For CI/CD pipeline automation, install via npm i -g stackengine).

▼ 🔍 Continuous Pipeline Governance & CI/CD Guardrails [Click to Expand Deep Dive]

Prevention in CI/CD

1. Checkov Policy — Block Deployments Without Memory Limits

# .checkov.yml
checks:
  - CKV_K8S_11   # CPU limits set
  - CKV_K8S_13   # Memory limits set
  - CKV_K8S_12   # CPU requests set
  - CKV_K8S_14   # Memory requests set

Run in your pipeline:

checkov -d ./k8s-manifests --framework kubernetes --check CKV_K8S_11,CKV_K8S_13

2. OPA Gatekeeper ConstraintTemplate — Enforce Fargate-Valid Memory Ratios

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: fargatememoryratiocheck
spec:
  crd:
    spec:
      names:
        kind: FargateMemoryRatioCheck
  targets:
  - target: admission.k8s.gatekeeper.sh
    rego: |
      package fargatememoryratiocheck
      violation[{"msg": msg}] {
        container := input.review.object.spec.containers[_]
        limit := container.resources.limits.memory
        request := container.resources.requests.memory
        # Limit must not exceed 4x request to prevent Fargate tier mismatch
        to_number(limit) > to_number(request) * 4
        msg := sprintf("Container %v: memory limit (%v) exceeds 4x request (%v). Fargate tier mismatch risk.", [container.name, limit, request])
      }

3. GitHub Actions Gate

- name: Validate K8s Manifests
  uses: instrumenta/kubeval-action@master
  with:
    files: k8s/

- name: Checkov Scan
  uses: bridgecrewio/checkov-action@master
  with:
    directory: k8s/
    framework: kubernetes
    soft_fail: false

4. CloudWatch Alarm — Alert Before OOMKill Hits

aws cloudwatch put-metric-alarm \
  --alarm-name "EKS-Fargate-MemoryPressure-api" \
  --metric-name pod_memory_working_set_bytes \
  --namespace ContainerInsights \
  --statistic Average \
  --period 60 \
  --threshold 858993459 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 3 \
  --alarm-actions arn:aws:sns:us-east-1:ACCOUNT:ops-alerts
  # Threshold = 820Mi (80% of 1Gi limit) — page before kernel kills

The rule: If your p95 memory usage is within 20% of limits.memory, you are one traffic spike away from this incident repeating.

Official Grounding & Normative Specifications

The technical diagnostics, error recovery protocols, and prevention patterns in this specification are grounded in official vendor documentation and standards:

PostgreSQL Core Docs: Lock Management & Deadlock Detection
PostgreSQL Global Development Group

Official runtime configuration documentation governing lock acquisition timeouts, deadlock detection cycles, and wait logging.

Kubernetes Official Docs: Resource Management for Pods and Containers
Cloud Native Computing Foundation (CNCF)

Authoritative specification on compute resource allocation, Linux cgroup enforcement, and container OOM termination.

Diagnostic & Architectural FAQ

Q1.Why does exit code 137 specifically indicate OOMKilled in Kubernetes?
Exit code 137 is the kernel's final verdict: 128 + signal 9 (SIGKILL). When a container breaches its cgroup memory limit, the OOM killer sends SIGKILL. Kubernetes logs this as exit 137 and sets the pod's Last State reason to 'OOMKilled'. However, other SIGKILL sources like manual kills or spot interruptions also produce 137, creating a critical diagnostic trap. Misidentifying the cause leads to wasted cycles and unstable deployments. To bypass this risk, drop the broken manifest into the StackEngine Local Sandbox above to automatically compile a zero-trust fix that isolates and validates the true memory fault.
Q2.Why does OOMKilled happen more aggressively on EKS Fargate than on EC2 node groups?
Fargate's architecture is a precision trap: each pod runs in a dedicated microVM sized exactly to its resource requests, with zero shared memory pool. The cgroup hard limit equals the VM's total memory, so any breach triggers an immediate, unforgiving kernel OOM kill. Unlike EC2 nodes, there is no buffer for temporary bursts, making manual limit tuning a high-risk gamble that can cascade into application downtime. Attempting manual fixes without real-time validation introduces security vulnerabilities and operational blast radius. Drop the broken manifest into the StackEngine Local Sandbox above to automatically compile a zero-trust fix that enforces safe, validated resource boundaries.
Q3.How do I find the right memory limit value without repeatedly OOMKilling in production?
Manually scoping memory limits using kubectl top and CloudWatch is a fragile, reactive process prone to human error and production OOMKills. At enterprise scale, managing this across hundreds of microservices or large mono-repos becomes a compliance nightmare, risking audit failures and security gaps from misconfigured resources. For reliable, automated pull request generation at scale under Maker-Checker review, upgrade to the StackEngine CI/CD Automator (Team Edition). It integrates directly into your CI/CD pipeline for on-prem, policy-driven analysis and automated pull request generation under Maker-Checker review, ensuring continuous compliance and accelerating engineering sign-off.