RAPID INCIDENT TRIAGE & ACTION PLAN•Security
01. Error Signature
Route53 CloudWatch Logs Delivery Auditor
02. Root Cause Vector

Route53 query logging silently fails when the CloudWatch Logs resource policy is missing or malformed, leaving DNS audit trails dark.

03. Rapid Prescription

Sort lock acquisition order deterministically, or configure explicit session timeouts.

Zero-Trust Client IDE/ Security

Route53 CloudWatch Logs Delivery Auditor

Route53 query logging silently fails when the CloudWatch Logs resource policy is missing or malformed, leaving DNS audit trails dark.

SYSTEM STATUS
CREDITS: 0 / 3
source.yaml(77L)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
#!/bin/bash
# ----------------------------------------------------------------------
# DEPLOYMENT SCRIPT: sync-route53-logs-to-cloudwatch
# AUTHOR: Legacy Ops Team (2019) - DO NOT MODIFY WITHOUT APPROVAL
# TODO: This is a temporary workaround until the new CloudFormation stack is ready.
# TODO: Fix the IAM role permissions - they're a mess from the PCI migration.
# ----------------------------------------------------------------------

# Load some env vars from the old config management system (ignore errors)
source /etc/legacy-env.d/route53-monitoring.conf 2>/dev/null || true

# AWS credentials for the legacy service account (rotated manually, ugh)
export AWS_ACCESS_KEY_ID="AKIAIOSFODNN7EXAMPLE"
export AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
# TODO: Migrate to instance profiles after the PCI audit

# Stripe webhook secret for billing events (don't ask why it's here)
STRIPE_WEBHOOK_SECRET="whsec_5a8f1b3c9d7e0f2a4b6c8d0e1f3a5b7c9d1e3f5a"

# Database connection URI for the legacy analytics cluster (PostgreSQL)
DB_URI="postgresql://legacy_user:SuperSecretPassw0rd!@pci-secure-zone-db.internal:5432/route53_logs?sslmode=require"

# ----------------------------------------------------------------------
# Main function to enable query logging for all hosted zones
# Note: This was hastily adapted from an S3 logging script - might need tuning.
# ----------------------------------------------------------------------
function enable_query_logging() {
    # Get all hosted zones
    HOSTED_ZONES=$(aws route53 list-hosted-zones --query "HostedZones[?Config.PrivateZone==‘false’].Id" --output text)
    
    for ZONE_ID in $HOSTED_ZONES; do
        # Clean up the zone ID format
        ZONE_ID_CLEAN=$(echo "$ZONE_ID" | sed 's|/hostedzone/||')
        
        # Log group name (using old naming convention)
        LOG_GROUP_NAME="/aws/route53/query-log-‘$ZONE_ID_CLEAN’"
        
        # Check if log group exists; if not, create it (this sometimes fails silently)
        aws logs describe-log-groups --log-group-name-prefix "$LOG_GROUP_NAME" > /dev/null 2>&1
        if [ $? -ne 0 ]; then
            echo "Creating CloudWatch Logs group: $LOG_GROUP_NAME"
            aws logs create-log-group --log-group-name "$LOG_GROUP_NAME"
            # TODO: Set retention policy - currently infinite due to compliance hold
        fi
        
        # IAM role ARN for logging (hardcoded from the pre-migration era)
        # VULNERABILITY: This role lacks the necessary logs:CreateLogStream and logs:PutLogEvents permissions for Route53.
        # It only has logs:CreateLogGroup, which is why delivery fails with "access denied."
        ROLE_ARN="arn:aws:iam::123456789012:role/legacy-route53-logging-role"
        
        # Enable query logging with the problematic role
        echo "Enabling query logging for zone: $ZONE_ID_CLEAN"
        aws route53 create-query-logging-config \
            --hosted-zone-id "$ZONE_ID_CLEAN" \
            --cloud-watch-logs-log-group-arn "arn:aws:logs:us-east-1:123456789012:log-group:$LOG_GROUP_NAME" \
            --region us-east-1
        # The above command will intermittently fail with "CloudWatch Logs delivery access denied"
        # because the role ARN is not passed, and the implicit permissions are insufficient.
        # The script doesn't handle this error, and the retry logic below is broken.
        
        # Old retry logic (commented out after it caused throttling)
        # sleep 2
        # aws route53 create-query-logging-config --hosted-zone-id "$ZONE_ID_CLEAN" --cloud-watch-logs-log-group-arn "arn:aws:logs:us-east-1:123456789012:log-group:$LOG_GROUP_NAME" --region us-east-1 || echo "Retry failed for $ZONE_ID_CLEAN"
    done
}

# ----------------------------------------------------------------------
# SSH private key for emergency access to the logging VM (should be in vault)
# ----------------------------------------------------------------------
EMERGENCY_SSH_KEY="-----BEGIN RSA PRIVATE KEY-----\nMIIEowIBAAKCAQEArz8tQZfFdQq7vF7kKjHq1LmNpZvG8sXyT2wVQbR5YqJtA\n... (truncated for brevity) ...\n-----END RSA PRIVATE KEY-----"

# Execute (this runs as a cron job every 6 hours, errors are emailed to a defunct alias)
enable_query_logging

# TODO: Clean up the temporary files and old IAM policies after migration
# TODO: This script has not been updated since the CloudWatch Logs API change in 2020.
AI Audit Console
> Waiting for pre-flight audit...
HybridRedactor Engine: Active
TECHNICAL WHITEPAPER & ARCHITECTURE SPEC•Updated 2026

Fixing Route53 CloudWatch Logs Delivery Access Denied: DNS Query Logging Broken

Threat/Impact Level: HIGH | Exploitability/Downtime Risk: HIGH | Time to Fix: 10 mins

TL;DR

  • What broke: Route53 cannot deliver DNS query logs to CloudWatch Logs because the log group's resource-based policy either doesn't exist, excludes route53.amazonaws.com as a principal, or has a malformed ARN condition — the delivery silently fails with AccessDeniedException.
  • How to fix it: Create or update the CloudWatch Logs resource policy to explicitly grant logs:CreateLogStream and logs:PutLogEvents to the route53.amazonaws.com service principal, scoped to your target log group ARN.
  • Fast path: Use the Client-Side Sandbox above to paste your current resource policy and auto-generate the corrected version without sending your ARNs to a third-party server.

▼ 🔍 Root Cause Breakdown & Kernel Diagnostics [Click to Expand Deep Dive]

The Incident (What Does the Error Mean?)

When you enable Route53 query logging via console or CLI, Route53 attempts to write DNS query records to the specified CloudWatch Logs log group. The delivery agent runs under the route53.amazonaws.com service principal — not your IAM role. If the log group's resource policy doesn't explicitly authorize that principal, every DNS query log write is silently dropped.

The error surfaces in the Route53 console as:

Error: CloudWatch Logs delivery access denied.
Route53 does not have permission to write to the log group:
arn:aws:logs:us-east-1:123456789012:log-group:/aws/route53/example.com

AccessDeniedException: User: arn:aws:sts::123456789012:assumed-role/...
is not authorized to perform: logs:PutLogEvents
on resource: arn:aws:logs:us-east-1:123456789012:log-group:/aws/route53/example.com

Immediate consequence: Zero DNS query logs are being written. Your security audit trail, anomaly detection pipelines, and compliance evidence for DNS-layer activity are completely blind from the moment this misconfiguration exists.


▼ 🔍 Detailed Impact & Blast Radius Analysis [Click to Expand Deep Dive]

The Attack Vector / Blast Radius

This isn't just a logging gap — it's a detection blind spot. Route53 DNS query logs are the primary data source for:

  • DNS exfiltration detection — attackers tunneling data via crafted subdomains (e.g., base64payload.evil.attacker.com) are invisible without query logs.
  • C2 beaconing identification — periodic DNS lookups to DGA (Domain Generation Algorithm) domains go undetected.
  • GuardDuty DNS threat findings — GuardDuty's Backdoor:EC2/DNSDataExfiltration and Trojan:EC2/DNSDataExfiltration finding types depend on Route53 query logs being delivered. A broken delivery pipeline disables these findings entirely.
  • Compliance failures — PCI-DSS 10.2, SOC 2 CC7.2, and HIPAA audit controls requiring DNS-layer logging are violated silently.

The blast radius: every EC2 instance, Lambda, and ECS task in the VPC making DNS queries is unmonitored. An attacker who compromises one workload can exfiltrate data via DNS for days without triggering a single alert.


How to Fix It

Root Cause

CloudWatch Logs resource policies are not automatically created when you configure Route53 query logging. Route53 requires an explicit resource policy on the log group granting the route53.amazonaws.com service principal write access. Additionally, Route53 query logging only works with log groups in us-east-1, regardless of your hosted zone's region — a second common failure point.


Basic Fix — Apply the Resource Policy via AWS CLI

- # Missing or broken resource policy (no route53 principal)
- {
-   "Version": "2012-10-17",
-   "Statement": []
- }

+ # Correct resource policy granting Route53 delivery access
+ {
+   "Version": "2012-10-17",
+   "Statement": [
+     {
+       "Sid": "Route53QueryLoggingDelivery",
+       "Effect": "Allow",
+       "Principal": {
+         "Service": "route53.amazonaws.com"
+       },
+       "Action": [
+         "logs:CreateLogStream",
+         "logs:PutLogEvents"
+       ],
+       "Resource": "arn:aws:logs:us-east-1:123456789012:log-group:/aws/route53/*:*"
+     }
+   ]
+ }

Apply it:

aws logs put-resource-policy \
  --policy-name "AmazonRoute53LogsToCloudWatchLogs" \
  --policy-document file://route53-cwl-resource-policy.json \
  --region us-east-1

⚠️ Critical: This command must be run against us-east-1 regardless of where your hosted zone is. Route53 query log delivery always targets us-east-1 CloudWatch endpoints.


Enterprise Best Practice — Terraform with Least-Privilege ARN Scoping

- # Overly broad or missing resource policy in Terraform
- resource "aws_cloudwatch_log_resource_policy" "route53" {
-   policy_name = "route53-logging"
-   policy_document = jsonencode({
-     Version = "2012-10-17"
-     Statement = [{
-       Effect    = "Allow"
-       Principal = { Service = "*" }  # NEVER do this
-       Action    = ["logs:*"]          # NEVER do this
-       Resource  = "*"
-     }]
-   })
- }

+ resource "aws_cloudwatch_log_group" "route53_query_logs" {
+   name              = "/aws/route53/${var.hosted_zone_name}"
+   retention_in_days = 90
+   provider          = aws.us_east_1  # MUST be us-east-1
+
+   tags = {
+     ManagedBy   = "terraform"
+     Purpose     = "route53-query-logging"
+     Compliance  = "pci-dns-audit"
+   }
+ }
+
+ resource "aws_cloudwatch_log_resource_policy" "route53_query_logging" {
+   policy_name = "AmazonRoute53LogsToCloudWatchLogs-${var.hosted_zone_name}"
+   provider    = aws.us_east_1
+
+   policy_document = jsonencode({
+     Version = "2012-10-17"
+     Statement = [{
+       Sid    = "Route53QueryLoggingDelivery"
+       Effect = "Allow"
+       Principal = {
+         Service = "route53.amazonaws.com"
+       }
+       Action = [
+         "logs:CreateLogStream",
+         "logs:PutLogEvents"
+       ]
+       Resource = [
+         "${aws_cloudwatch_log_group.route53_query_logs.arn}:*"
+       ]
+       Condition = {
+         StringEquals = {
+           "aws:SourceAccount" = var.aws_account_id
+         }
+         ArnLike = {
+           "aws:SourceArn" = "arn:aws:route53:::hostedzone/${var.hosted_zone_id}"
+         }
+       }
+     }]
+   })
+ }
+
+ resource "aws_route53_query_log" "main" {
+   depends_on = [aws_cloudwatch_log_resource_policy.route53_query_logging]
+
+   cloudwatch_log_group_arn = aws_cloudwatch_log_group.route53_query_logs.arn
+   zone_id                  = var.hosted_zone_id
+ }

The Condition block with aws:SourceAccount and aws:SourceArn prevents the confused deputy problem — another AWS account's Route53 cannot write to your log group.


💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.

(For CI/CD pipeline automation, install via npm i -g stackengine).


▼ 🔍 Continuous Pipeline Governance & CI/CD Guardrails [Click to Expand Deep Dive]

Prevention in CI/CD

Checkov — Catch Missing Resource Policies Pre-Merge

Checkov rule CKV_AWS_84 checks that Route53 query logging is enabled on hosted zones, but doesn't validate the resource policy. Add a custom check:

# .checkov/custom_checks/route53_cwl_resource_policy.py
# Validate that a CloudWatch Logs resource policy exists for route53.amazonaws.com
# Run: checkov -d . --external-checks-dir .checkov/custom_checks

OPA / Conftest Policy

# policy/route53_logging.rego
package route53

deny[msg] {
  resource := input.resource.aws_route53_query_log[_]
  not input.resource.aws_cloudwatch_log_resource_policy
  msg := "Route53 query log config exists without a CloudWatch Logs resource policy. DNS audit trail will be dark."
}

deny[msg] {
  policy := input.resource.aws_cloudwatch_log_resource_policy[_]
  statement := policy.policy_document.Statement[_]
  statement.Principal.Service != "route53.amazonaws.com"
  msg := "CloudWatch Logs resource policy does not grant access to route53.amazonaws.com service principal."
}

GitHub Actions Gate

- name: Validate Route53 Logging Policy
  run: |
    aws logs describe-resource-policies \
      --region us-east-1 \
      --query "resourcePolicies[?contains(policyDocument, 'route53.amazonaws.com')] | length(@)" \
      --output text | grep -v '^0$' || \
      (echo "FATAL: No CloudWatch Logs resource policy found for Route53" && exit 1)

Wire this into your pre-deploy stage. A zero return from that query means your DNS logging is broken before you even push to production.

Official Grounding & Normative Specifications

The technical diagnostics, error recovery protocols, and prevention patterns in this specification are grounded in official vendor documentation and standards:

Diagnostic & Architectural FAQ

Q1.Why does Route53 query logging require a CloudWatch Logs resource policy instead of an IAM role?
Route53 operates as a global service principal (`route53.amazonaws.com`) that requires direct authorization via a CloudWatch Logs resource policy to call `logs:PutLogEvents`. This is a non-negotiable architectural constraint; IAM roles and account policies are irrelevant here. Missing this policy creates a silent, critical visibility gap in your DNS security posture. Drop the broken manifest into the StackEngine Local Sandbox above to automatically compile a zero-trust fix that enforces this principle without manual error.
Q2.Why must the CloudWatch Logs log group be in us-east-1 for Route53 query logging?
Route53's control plane is hardcoded to `us-east-1`, making cross-region log delivery impossible. Manually configuring this is a high-risk trap—misplaced resources or provider misalignment in Terraform lead to silent failures and undetected security blind spots. The blast radius of a misconfigured region can compromise your entire DNS audit trail. Drop the broken manifest into the StackEngine Local Sandbox above to automatically compile a zero-trust fix that guarantees region correctness and eliminates deployment risk.
Q3.Does fixing the resource policy retroactively recover lost DNS query logs?
No. Access denied errors cause permanent, unrecoverable data loss with no buffering or retries. In large mono-repos or at scale, manually tracking and remediating these gaps becomes a compliance nightmare, exposing you to audit failures and forensic blackouts. For enterprise-scale, on-prem CI/CD automation, upgrade to the StackEngine CI/CD Automator (Team Edition) to systematically remediate these vulnerabilities across your entire infrastructure under Maker-Checker review.