K8s Secret 1MB Payload Size Validator

The Incident (What Does the Error Mean?)

Raw error from kubectl:

Error from server: etcdserver: request is too large

or, depending on API server version:

The Secret "my-app-secret" is invalid: []: Too long: must have at most 1048576 bytes

Immediate consequence: The Secret object never lands in etcd. Any Deployment, StatefulSet, or Job that mounts this Secret stays in Pending or CreateContainerConfigError. Your rollout is dead until this is resolved. There is no partial write — the entire object is rejected atomically.

The 1MB ceiling is not a Kubernetes configuration knob. It is enforced by etcd's default --max-request-bytes (1,572,864 bytes at the etcd layer, but the Kubernetes API server imposes a tighter 1,048,576-byte limit on individual object values). Base64 encoding inflates binary payloads by ~33%, meaning a 768KB binary blob becomes a ~1.02MB base64 string and trips the limit.

The Attack Vector / Blast Radius

This is primarily a stability and pipeline failure issue, but it has a security dimension that gets overlooked under incident pressure:

1. The workaround trap: Engineers under pressure to restore service frequently move the oversized payload into a ConfigMap (which has the same 1MB limit but feels "less restricted") or — worse — bake the secret directly into the container image. Baking secrets into images is a critical supply-chain vulnerability: the secret is now in your registry, your build cache, and every layer history.

2. etcd blast radius: If you somehow bypass the API server limit (e.g., via a direct etcd write during a migration), a single oversized object can cause etcd compaction pressure and slow all control-plane reads for the entire cluster. Leader election for kube-scheduler and kube-controller-manager degrades. At scale, this cascades into node NotReady events.

3. Secret sprawl under pressure: Splitting a secret incorrectly — e.g., storing part of a TLS chain in a Secret and the key in a ConfigMap — creates a split-trust boundary that breaks your audit trail and may violate PCI-DSS / SOC2 secret handling requirements.

Prevention in CI/CD

1. OPA/Gatekeeper Policy (blocks oversized Secrets at admission)

package kubernetes.secret.sizelimit

violation[{"msg": msg}] {
  input.review.object.kind == "Secret"
  total := sum([count(v) | v := input.review.object.data[_]])
  total > 786432  # 768KB base64 = ~512KB raw; hard block before hitting API limit
  msg := sprintf("Secret '%v' total data size %v bytes exceeds 768KB policy limit. Externalize large payloads.", [
    input.review.object.metadata.name, total
  ])
}

2. Checkov / Pre-commit Hook

Checkov does not have a built-in check for this. Add a custom check or use this shell gate in your CI pipeline:

#!/usr/bin/env bash
# ci/validate-secret-size.sh
MAX_BYTES=786432
FAILED=0
for f in $(find . -name '*.yaml' -o -name '*.yml'); do
  if grep -q 'kind: Secret' "$f"; then
    SIZE=$(python3 -c "
import yaml, base64, sys
with open('$f') as fh:
    doc = yaml.safe_load(fh)
if doc and doc.get('kind') == 'Secret':
    total = sum(len(v) for v in (doc.get('data') or {}).values())
    print(total)
else:
    print(0)
")
    if [ "$SIZE" -gt "$MAX_BYTES" ]; then
      echo "[FAIL] $f: Secret data is ${SIZE} bytes (limit: ${MAX_BYTES})"
      FAILED=1
    fi
  fi
done
exit $FAILED

3. Terraform (if managing Secrets via `kubernetes_secret` resource)

- resource "kubernetes_secret" "app_tls" {
-   data = {
-     "bundle.p12" = filebase64("${path.module}/certs/bundle.p12")  # No size guard
-   }
- }

+ locals {
+   p12_b64      = filebase64("${path.module}/certs/bundle.p12")
+   p12_size     = length(local.p12_b64)
+ }
+
+ resource "null_resource" "secret_size_guard" {
+   lifecycle {
+     precondition {
+       condition     = local.p12_size < 786432
+       error_message = "bundle.p12 base64 size ${local.p12_size} exceeds 768KB CI policy. Use Vault or split the payload."
+     }
+   }
+ }
+
+ resource "kubernetes_secret" "app_tls" {
+   depends_on = [null_resource.secret_size_guard]
+   data = {
+     "bundle.p12" = local.p12_b64
+   }
+ }

Bottom line: The 1MB etcd limit is immovable in standard clusters. Enforce a 768KB soft ceiling in CI so you have 25% headroom for Secret metadata overhead and never hit the hard wall in production again.