Initializing Enclave...

How to Fix 'Provider Produced Inconsistent Final Plan' for aws_instance in Terraform

Threat/Impact Level: HIGH | Downtime Risk: HIGH | Time to Fix: 15–45 mins

TL;DR

  • What broke: Terraform's AWS provider generated a plan, then re-evaluated computed attributes (AMI ID, user_data hash, tags) during the apply phase and got a different result — a hard fatal mismatch that kills the apply mid-run.
  • How to fix it: Pin the offending computed value to a data source or local, remove it from lifecycle.ignore_changes if it's not truly static, and lock your AWS provider to a patch version that doesn't have known plan-diff bugs.
  • Fast path: Use our Client-Side Sandbox below to auto-refactor this — paste your aws_instance block and get the corrected HCL without leaking your AMI IDs or account numbers.

The Incident (What Does the Error Mean?)

You hit this during terraform apply:

Error: Provider produced inconsistent final plan

When expanding the plan for aws_instance.app_server to include new values
learned so far during apply, provider "registry.terraform.io/hashicorp/aws"
produced an invalid new value for .ami: was cty.StringVal("ami-0abcdef1234567890"),
but now cty.StringVal("ami-0zyxwvu9876543210").

This is a bug in the provider, which should be reported in the provider's
own issue tracker.

Immediate consequence: The apply is aborted. If this was a create, no instance exists. If this was an update on a running instance, Terraform may have already destroyed the old one depending on create_before_destroy settings — leaving you in a partial state with potential data loss or service outage.

The core mechanic: Terraform runs planning in two passes. Pass 1 (the plan phase) resolves known values. Pass 2 (the apply phase) re-resolves them with live API data. If the provider returns a different value in pass 2 — because a data source re-evaluated, a function produced non-deterministic output, or the provider itself has a bug — Terraform panics with this error rather than silently applying the wrong config.

The Blast Radius

This isn't just an inconvenience. The blast radius depends on your pipeline:

Scenario A — New Instance (Create): Apply fails before the instance is created. Clean failure. Annoying but safe.

Scenario B — Replace on Existing Instance: If create_before_destroy = true is set, Terraform may have already launched the new instance before detecting the inconsistency on the old one's teardown. You now have a ghost instance accumulating cost with no state entry.

Scenario C — CI/CD Auto-Apply: An automated pipeline running terraform apply -auto-approve hits this, logs the error, and exits non-zero. If your pipeline has poor rollback logic, your infrastructure state is now diverged from reality. Every subsequent plan will be dirty.

Scenario D — user_data Drift: If the inconsistency is in user_data, the instance bootstrap script is non-deterministic. The instance may launch with the wrong startup configuration — wrong environment variables, wrong S3 bucket paths, wrong IAM role assumptions baked in at boot. This is a silent misconfiguration that won't show up in logs until the app fails at runtime.

How to Fix It

Root Cause Checklist

Before touching code, identify which attribute is drifting. The error message tells you directly. Common offenders:

Drifting Attribute Why It Drifts
ami data.aws_ami with most_recent = true returns a new AMI between plan and apply
user_data Template function or external file read is non-deterministic
tags merge() with a computed map that resolves differently at apply time
vpc_security_group_ids Security group created in same apply, ID not yet known at plan time
iam_instance_profile Profile ARN computed from a resource created in the same root module

Basic Fix — Pin the AMI (Most Common Case)

The most frequent cause is data.aws_ami with most_recent = true. Between terraform plan and terraform apply, AWS releases a new AMI version. The data source re-queries and returns a different ID.

- data "aws_ami" "app" {
-   most_recent = true
-   owners      = ["amazon"]
-
-   filter {
-     name   = "name"
-     values = ["amzn2-ami-hvm-*-x86_64-gp2"]
-   }
- }
-
- resource "aws_instance" "app_server" {
-   ami           = data.aws_ami.app.id
-   instance_type = "t3.medium"
- }
+ # Pin to a specific, tested AMI ID via variable or locals
+ # Resolve the AMI ID ONCE in your pipeline (e.g., Packer output)
+ # and pass it as a variable. Never let most_recent=true run at apply time
+ # in automated pipelines.
+
+ variable "app_ami_id" {
+   type        = string
+   description = "Pinned AMI ID validated by Packer pipeline. Update per release."
+   # default   = "ami-0abcdef1234567890"  # set via tfvars or CI env var
+ }
+
+ resource "aws_instance" "app_server" {
+   ami           = var.app_ami_id
+   instance_type = "t3.medium"
+ }

Basic Fix — Stabilize user_data

- resource "aws_instance" "app_server" {
-   ami           = var.app_ami_id
-   instance_type = "t3.medium"
-
-   # templatefile() re-reads from disk on every evaluation pass
-   # If the file or vars change between plan/apply, you get drift
-   user_data = templatefile("${path.module}/scripts/init.sh.tpl", {
-     db_host = aws_db_instance.main.address
-   })
- }
+ locals {
+   # Compute user_data ONCE into a local. Terraform evaluates locals
+   # deterministically within a single operation.
+   user_data_rendered = templatefile("${path.module}/scripts/init.sh.tpl", {
+     db_host = aws_db_instance.main.address
+   })
+ }
+
+ resource "aws_instance" "app_server" {
+   ami           = var.app_ami_id
+   instance_type = "t3.medium"
+   user_data     = local.user_data_rendered
+
+   # Prevent replacement on user_data changes to existing instances
+   # unless you explicitly intend a rolling replacement
+   lifecycle {
+     ignore_changes = [user_data]
+   }
+ }

Enterprise Best Practice — Full Production-Grade aws_instance Block

- resource "aws_instance" "app_server" {
-   ami                  = data.aws_ami.app.id  # non-deterministic
-   instance_type        = "t3.medium"
-   iam_instance_profile = aws_iam_instance_profile.app.name
-
-   tags = merge(var.common_tags, {
-     Name = "app-server-${random_id.suffix.hex}"  # computed at apply
-   })
-
-   lifecycle {
-     ignore_changes = [ami, tags]  # masking the real problem
-   }
- }
+ locals {
+   # All computed values resolved once, referenced by resource
+   instance_name     = "app-server-${var.environment}-${var.region_short}"
+   user_data_hash    = sha256(local.user_data_rendered)  # for change detection
+   user_data_rendered = templatefile("${path.module}/scripts/init.sh.tpl", {
+     db_host    = aws_db_instance.main.address
+     app_bucket = aws_s3_bucket.app_data.bucket
+   })
+ }
+
+ resource "aws_instance" "app_server" {
+   ami                    = var.app_ami_id  # pinned, passed from CI/Packer
+   instance_type          = var.instance_type
+   iam_instance_profile   = aws_iam_instance_profile.app.name
+   subnet_id              = var.private_subnet_id
+   vpc_security_group_ids = [aws_security_group.app.id]
+   user_data              = local.user_data_rendered
+
+   # Structured tags — no computed values inline
+   tags = merge(var.common_tags, {
+     Name        = local.instance_name
+     Environment = var.environment
+     ManagedBy   = "terraform"
+   })
+
+   # Explicit replacement strategy — no silent drift
+   lifecycle {
+     create_before_destroy = true
+     # Only ignore user_data if bootstrap is idempotent and
+     # you handle config via SSM/Ansible post-boot
+     ignore_changes = [user_data]
+   }
+
+   # Ensure dependent resources exist before instance launches
+   depends_on = [
+     aws_iam_role_policy_attachment.app_ssm,
+     aws_security_group.app
+   ]
+ }

Provider Version Bug Path

If your attribute is not in the checklist above and the error points to a computed attribute you don't control (like private_dns, private_ip, or availability_zone), you're likely hitting a known provider bug.

 terraform {
   required_providers {
     aws = {
       source  = "hashicorp/aws"
-      version = "~> 5.0"   # too loose — picks up buggy patch releases
+      version = "= 5.31.0" # pin to last known-good version
     }
   }
 }

Check the AWS provider CHANGELOG for your version. Search for "inconsistent" or the specific attribute name.

💡 Tired of pasting proprietary configs into ChatGPT? Generic AI tools log your company's ARNs, DB strings, and private keys. StackEngine is a zero-backend, pure Client-Side WASM utility. Drop your failing config into the sandbox above. We redact your secrets locally in the browser and auto-generate the refactored code using your own API key.

Prevention in CI/CD

1. Enforce Deterministic Plans with -refresh=false in Pipelines

For apply pipelines where you've already run plan and saved the plan file, use the saved plan:

# In CI: plan stage
terraform plan -out=tfplan.binary

# In CI: apply stage — applies EXACTLY the saved plan, no re-evaluation
terraform apply tfplan.binary

This is the single most effective mitigation. The apply stage cannot drift if it's consuming a binary plan artifact.

2. Checkov — Detect Non-Deterministic AMI Lookups

# .checkov.yml
checks:
  - CKV_AWS_8   # Ensures detailed monitoring is enabled
  # Add custom check for most_recent=true in data sources
custom_checks_dir: ./checkov_custom

# checkov_custom/check_ami_pinned.py
from checkov.terraform.checks.resource.base_resource_check import BaseResourceCheck
from checkov.common.models.enums import CheckResult, CheckCategories

class AmiPinnedCheck(BaseResourceCheck):
    def __init__(self):
        name = "Ensure AMI is pinned via variable, not most_recent data source"
        id = "CKV_CUSTOM_AMI_001"
        supported_resources = ["aws_instance"]
        categories = [CheckCategories.GENERAL_SECURITY]
        super().__init__(name=name, id=id, categories=categories,
                         supported_resources=supported_resources)

    def scan_resource_conf(self, conf):
        ami = conf.get("ami", [""])
        # Flag if AMI value references a data source (most_recent pattern)
        if isinstance(ami, list) and "data.aws_ami" in str(ami[0]):
            return CheckResult.FAILED
        return CheckResult.PASSED

scanner = AmiPinnedCheck()

3. OPA/Conftest Policy — Block most_recent = true in Plans

# policies/no_dynamic_ami.rego
package terraform.aws.instance

deny[msg] {
  resource := input.resource_changes[_]
  resource.type == "aws_instance"
  # Check if AMI value is unknown/computed at plan time (null in plan JSON)
  resource.change.after.ami == null
  msg := sprintf(
    "aws_instance '%s': AMI must be a pinned, known value at plan time. Use var.ami_id.",
    [resource.address]
  )
}

# In CI pipeline
terraform show -json tfplan.binary > tfplan.json
conftest test tfplan.json --policy policies/

4. terraform validate + tflint Pre-Commit Hook

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/antonbabenko/pre-commit-terraform
    rev: v1.88.0
    hooks:
      - id: terraform_validate
      - id: terraform_tflint
        args:
          - --args=--enable-rule=terraform_required_version
          - --args=--enable-rule=aws_instance_invalid_ami

5. Lock Provider Versions in Every Module

# versions.tf — in EVERY module, not just root
terraform {
  required_version = ">= 1.6.0, < 2.0.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "= 5.31.0"  # exact pin, reviewed and bumped deliberately
    }
  }
}

Commit your .terraform.lock.hcl to version control. Never .gitignore it.

Related Diagnostics

PostgreSQL JSONB UTF8 Byte Sequence Fixer
Analyze →
Next.js Default Export Component Validator
Analyze →

"Part of the Syntax Utility Matrix."

View all 153 Syntax Tools →