Etcd Large ConfigMap Payload Optimizer

The Incident (What Does the Error Mean?)

Error from server: etcdserver: request is too large

or in etcd logs:

WARN etcdserver/v3_server.go:xxx rejected request, too large size=1612800 max=1572864

etcd enforces a hard ceiling on individual RPC payload size. The default is 1572864 bytes (1.5MiB). The Kubernetes API server serializes the entire ConfigMap object — data, binaryData, metadata, annotations — into a single etcd Put request. If that serialized blob exceeds the limit, etcd drops it at the transport layer. The API server propagates this as a generic error. Your Deployment, StatefulSet, or Helm release is now stuck. Any controller waiting on that ConfigMap version will loop or crash-backoff.

The Attack Vector / Blast Radius

This is a control plane availability failure, not a data-plane issue. The blast radius:

1. Rollout freeze: Any workload referencing the oversized ConfigMap via envFrom or volumeMount will fail to reconcile. Kubernetes controllers will retry on exponential backoff, generating noise that masks other real errors.
2. etcd compaction pressure: Repeated failed large writes still generate WAL entries in some etcd versions before the reject gate. Under sustained retry storms, this accelerates disk I/O and compaction cycles.
3. API server latency spike: The API server serializes the full object before the etcd rejection. In high-traffic clusters (>50 req/s to the ConfigMap endpoint), this serialization overhead on every retry contributes to API server p99 latency degradation — other unrelated workloads feel it.
4. Helm/ArgoCD deadlock: Helm stores release state in Secrets, but if your Helm chart includes a ConfigMap that triggers this error, helm upgrade will partially apply resources and leave the release in a pending-upgrade state. Manual intervention required.
5. Raising --max-request-bytes is not a fix — it raises the ceiling on every etcd node, increases snapshot size, slows leader elections, and pushes the problem to a higher threshold you'll hit again.

Prevention in CI/CD

1. OPA/Gatekeeper Policy — Block Oversized ConfigMaps at Admission

# ConstraintTemplate: deny ConfigMaps exceeding 512KB
package kubernetes.configmap.sizelimit

violation[{"msg": msg}] {
  input.review.object.kind == "ConfigMap"
  total := sum([count(v) | v := input.review.object.data[_]])
  total > 524288
  msg := sprintf("ConfigMap data exceeds 512KB limit (%d bytes). Split or externalize large keys.", [total])
}

2. Pre-commit / CI Lint Step

# Add to your CI pipeline before kubectl apply
for f in $(find ./k8s -name '*.yaml'); do
  size=$(python3 -c "
import yaml, sys, json
with open('$f') as fh:
    docs = list(yaml.safe_load_all(fh))
for d in docs:
    if d and d.get('kind') == 'ConfigMap':
        total = sum(len(v.encode()) for v in (d.get('data') or {}).values())
        if total > 524288:
            print(f'FAIL: {f} ConfigMap data={total} bytes > 512KB limit')
            sys.exit(1)
")
done

3. Helm Chart Guardrails

- # values.yaml — no size validation
- configData: |
-   {{ .Files.Get "large-seed.json" | b64enc }}

+ # Add a Helm test or pre-install hook that validates size
+ # Use .Files.Glob and fail loudly in _helpers.tpl:
+ {{- if gt (len (.Files.Get "large-seed.json")) 524288 }}
+ {{- fail "large-seed.json exceeds 512KB. Use an init container or external store." }}
+ {{- end }}

4. Checkov Rule (IaC Scanning)

Checkov doesn't ship a native ConfigMap size check, but you can add a custom check:

# custom_checks/configmap_size.py
from checkov.common.models.enums import CheckResult
from checkov.kubernetes.checks.resource.base_resource_check import BaseK8Check

class ConfigMapSizeCheck(BaseK8Check):
    def __init__(self):
        super().__init__(name="ConfigMap data must not exceed 512KB",
                         check_id="CKV_K8S_CUSTOM_001",
                         supported_entities=['ConfigMap'])
    def scan_resource_conf(self, conf):
        data = conf.get('data') or {}
        total = sum(len(str(v).encode()) for v in data.values())
        return CheckResult.PASSED if total <= 524288 else CheckResult.FAILED

Key rule: Treat the 512KB mark as your internal limit — giving you 3x headroom before the etcd 1.5MB hard wall. By the time a ConfigMap hits 512KB in code review, it's already an architectural smell.