How to Fix Terraform 'for_each Set Contains Duplicate or Null' Error

Threat/Impact Level: HIGH | Exploitability/Downtime Risk: HIGH | Time to Fix: 10 mins

TL;DR

• What broke: Terraform's for_each received a set or map containing null values or duplicate keys, which is a hard type constraint violation — plan aborts immediately.
• How to fix it: Pipe your collection through compact() to strip nulls and distinct() to deduplicate before casting with toset().
• Fast path: Use our Client-Side Sandbox below to auto-refactor this — paste your broken for_each expression and get corrected HCL instantly.

---

The Incident (What Does the Error Mean?)

Raw error output from terraform plan:

Error: Invalid for_each argument

  on main.tf line 12, in resource "aws_iam_user" "this":
  12:   for_each = var.usernames

The given "for_each" argument value is unsuitable: "for_each" supports maps
and sets of strings, but you have provided a set containing null.

or the duplicate variant:

Error: Invalid for_each argument

The given "for_each" argument value is unsuitable: the set contains
duplicate values after applying type conversion.

Immediate consequence: Terraform refuses to build the execution graph. No resources are created, updated, or destroyed. Your pipeline is fully blocked. If this is in a module called across environments, every downstream workspace is dead.

---

The Attack Vector / Blast Radius

This isn't just a syntax annoyance. Here's the cascading failure chain:

1. Upstream data sources return nulls. A data.aws_ssm_parameter lookup for a non-existent key returns null. That null flows into a local list. That list feeds for_each. Boom.
2. Variable defaults are null. Optional variables typed as list(string) with default = null passed directly to for_each — common in module composition patterns.
3. toset() does not deduplicate before Terraform validates. If your list has ["us-east-1", "us-east-1"], toset() should collapse them, but if the underlying type coercion produces duplicates after conversion (e.g., mixed case strings coerced), Terraform throws before your code runs.
4. Blast radius in monorepos: A single broken module called 15 times across a Terragrunt stack means 15 broken plan runs, blocking your entire release pipeline.

The silent killer: This error frequently surfaces only in CI, not locally, because local terraform.tfvars has clean data while CI pulls from a secrets manager or remote state that can return nulls.

---

How to Fix It (The Solution)

Basic Fix — Strip Nulls and Deduplicate Inline

 resource "aws_iam_user" "this" {
-  for_each = var.usernames
+  for_each = toset(compact(distinct(var.usernames)))
   name     = each.key
 }

• distinct() — removes duplicate string values
• compact() — removes null and empty string "" values
• toset() — converts the clean list to the set type for_each requires

Order matters. Always distinct() → compact() → toset().

---

Enterprise Best Practice — Validate at the Variable Declaration

Don't fix it at the resource. Fix it at the boundary. Use validation blocks so the error is caught with a human-readable message before plan even starts:

 variable "usernames" {
   type = list(string)
+
+  validation {
+    condition     = length(var.usernames) == length(distinct(compact(var.usernames)))
+    error_message = "usernames must not contain null, empty strings, or duplicate values."
+  }
+
+  validation {
+    condition     = !contains(var.usernames, null)
+    error_message = "usernames list must not contain null values."
+  }
 }

And in the resource, still apply the defensive transform as a belt-and-suspenders approach:

 resource "aws_iam_user" "this" {
-  for_each = toset(var.usernames)
+  for_each = toset(compact(distinct(var.usernames)))
   name     = each.key
 }

For dynamic data from data sources (the most common production failure mode):

 locals {
-  target_arns = data.aws_ssm_parameter.endpoints[*].value
+  target_arns = toset(compact(distinct([
+    for v in data.aws_ssm_parameter.endpoints[*].value : v
+    if v != null && v != ""
+  ])))
 }

 resource "aws_lambda_event_source_mapping" "this" {
-  for_each        = toset(local.target_arns)
+  for_each        = local.target_arns
   event_source_arn = each.value
 }

---

💡 Tired of pasting proprietary configs into ChatGPT? Generic AI tools log your company's ARNs, DB strings, and private keys. StackEngine is a zero-backend, pure Client-Side WASM utility. Drop your failing config into the sandbox above. We redact your secrets locally in the browser and auto-generate the refactored code using your own API key.

---

Prevention in CI/CD

You should never see this error in a PR merge. Gate it earlier:

1. terraform validate in Pre-Commit

# .pre-commit-config.yaml
- repo: https://github.com/antonbabenko/pre-commit-terraform
  rev: v1.83.0
  hooks:
    - id: terraform_validate
    - id: terraform_tflint

2. TFLint Rule for Null Variables

Add to .tflint.hcl:

rule "terraform_required_providers" {
  enabled = true
}

plugin "aws" {
  enabled = true
  version = "0.27.0"
  source  = "github.com/terraform-linters/tflint-ruleset-aws"
}

TFLint will flag for_each expressions that reference variables without null guards.

3. Checkov Policy (OPA-compatible)

# checkov custom check — flag for_each without compact()
from checkov.terraform.checks.resource.base_resource_check import BaseResourceCheck

class ForEachNullGuardCheck(BaseResourceCheck):
    def __init__(self):
        name = "Ensure for_each uses compact() to prevent null set errors"
        id = "CKV_CUSTOM_TF_001"
        ...

4. CI Pipeline Gate (GitHub Actions)

- name: Terraform Validate
  run: |
    terraform init -backend=false
    terraform validate
  env:
    TF_VAR_usernames: '["placeholder"]'

Always inject a non-null placeholder value for list variables during validate runs — otherwise validate passes and plan fails.

---

Quick Reference: Null-Safe for_each Patterns

Input Type	Safe Pattern
list(string) variable	toset(compact(distinct(var.list)))
data source attribute list	toset([for v in data.x[*].attr : v if v != null])
Map with potential null values	{for k, v in var.map : k => v if v != null}
SSM / Secrets Manager output	Always use try(data.x.value, "") then compact()