EKS Fargate ENI IP Exhaustion Monitor

Threat/Impact Level: HIGH | Downtime Risk: HIGH | Time to Fix: 15–45 mins depending on subnet re-IP strategy

TL;DR

• What broke: Fargate can't allocate an ENI for the pod sandbox because every available IP in the configured subnet(s) is consumed. New pods are dead on arrival — FailedCreatePodSandBox in events, nothing scheduled.
• How to fix it: Add secondary CIDR blocks to your VPC, provision new larger subnets, update your Fargate profile to include them, and optionally enable prefix delegation on the VPC CNI.
• Shortcut: Use our Client-Side Sandbox above to auto-refactor your Fargate profile and Terraform VPC config — it detects exhausted CIDRs and outputs corrected subnet allocations without sending your ARNs anywhere.

Warning  FailedCreatePodSandBox  kubelet  Failed to create pod sandbox:
  rpc error: code = Unknown desc = failed to set up sandbox container
  "abc123" network for pod "my-service-7d9f8b-xkqzp":
  networkPlugin cni failed to set up pod
  "my-service-7d9f8b-xkqzp_production" network:
  add cmd: failed to assign an IP address to container

How to Fix It

Basic Fix — Add a New Subnet and Update the Fargate Profile

The fastest path out of a live outage: provision a new subnet in the same AZ with a larger CIDR, then add it to your Fargate profile.

# terraform/vpc.tf

- resource "aws_subnet" "fargate_primary" {
-   vpc_id            = aws_vpc.main.id
-   cidr_block        = "10.0.1.0/24"   # 251 IPs — exhausted
-   availability_zone = "us-east-1a"
- }

+ resource "aws_subnet" "fargate_primary" {
+   vpc_id            = aws_vpc.main.id
+   cidr_block        = "10.0.1.0/24"
+   availability_zone = "us-east-1a"
+ }
+
+ resource "aws_subnet" "fargate_overflow" {
+   vpc_id            = aws_vpc.main.id
+   cidr_block        = "10.0.16.0/20"  # 4091 IPs — room to grow
+   availability_zone = "us-east-1a"
+   tags = {
+     Name = "fargate-overflow-1a"
+   }
+ }

# terraform/fargate_profile.tf

 resource "aws_eks_fargate_profile" "default" {
   cluster_name           = aws_eks_cluster.main.name
   fargate_profile_name   = "default"
   pod_execution_role_arn = aws_iam_role.fargate_pod_execution.arn

   subnet_ids = [
-    aws_subnet.fargate_primary.id
+    aws_subnet.fargate_primary.id,
+    aws_subnet.fargate_overflow.id
   ]

   selector {
     namespace = "production"
   }
 }

⚠️ Fargate profile updates that change subnet_ids require profile recreation in some Terraform provider versions. Plan for a brief scheduling gap or use blue/green profile swap.

Enterprise Best Practice — Secondary CIDR + VPC CNI Prefix Delegation

If your VPC CIDR space is itself exhausted (common in large orgs with RFC 1918 carve-outs), attach a secondary CIDR from the 100.64.0.0/10 CG-NAT range (AWS supports this) and build Fargate subnets there.

# terraform/vpc.tf

 resource "aws_vpc" "main" {
   cidr_block = "10.0.0.0/16"
 }

+ resource "aws_vpc_ipv4_cidr_block_association" "cgnat_secondary" {
+   vpc_id     = aws_vpc.main.id
+   cidr_block = "100.64.0.0/16"  # CG-NAT range, AWS-supported secondary
+ }
+
+ resource "aws_subnet" "fargate_cgnat_1a" {
+   vpc_id            = aws_vpc.main.id
+   cidr_block        = "100.64.0.0/18"  # 16,379 IPs
+   availability_zone = "us-east-1a"
+   depends_on        = [aws_vpc_ipv4_cidr_block_association.cgnat_secondary]
+ }
+
+ resource "aws_subnet" "fargate_cgnat_1b" {
+   vpc_id            = aws_vpc.main.id
+   cidr_block        = "100.64.64.0/18"
+   availability_zone = "us-east-1b"
+   depends_on        = [aws_vpc_ipv4_cidr_block_association.cgnat_secondary]
+ }

Also add a CloudWatch alarm on available IPs so you never get blindsided again:

+ resource "aws_cloudwatch_metric_alarm" "subnet_ip_exhaustion" {
+   alarm_name          = "fargate-subnet-ip-low-${each.key}"
+   comparison_operator = "LessThanThreshold"
+   evaluation_periods  = 2
+   metric_name         = "AvailableIPAddresses"
+   namespace           = "AWS/EC2"
+   period              = 300
+   statistic           = "Minimum"
+   threshold           = 50   # Alert at 50 IPs remaining
+   dimensions = {
+     SubnetId = each.value
+   }
+   alarm_actions = [aws_sns_topic.ops_alerts.arn]
+ }

💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.

(For CI/CD pipeline automation, install via npm i -g stackengine).

# checkov custom check: enforce minimum /20 for Fargate subnets
from checkov.terraform.checks.resource.base_resource_check import BaseResourceCheck
import ipaddress

class FargateSubnetSizeCheck(BaseResourceCheck):
    ID = "CKV_CUSTOM_FARGATE_SUBNET_SIZE"
    NAME = "Fargate subnets must be /20 or larger"
    SUPPORTED_RESOURCES = ["aws_subnet"]
    BLOCK_TYPE = "resource"

    def scan_resource_conf(self, conf):
        cidr = conf.get("cidr_block", [""])[0]
        try:
            network = ipaddress.IPv4Network(cidr, strict=False)
            return network.prefixlen <= 20  # /20 or larger = PASS
        except ValueError:
            return False

# policy/fargate_subnet.rego
package fargate

deny[msg] {
  resource := input.resource_changes[_]
  resource.type == "aws_eks_fargate_profile"
  count(resource.change.after.subnet_ids) < 2
  msg := sprintf(
    "Fargate profile '%v' must reference subnets in at least 2 AZs to prevent single-AZ IP exhaustion.",
    [resource.name]
  )
}

- name: Conftest Fargate Policy Check
  run: |
    terraform show -json plan.tfplan > plan.json
    conftest test plan.json --policy policy/fargate_subnet.rego