CronJob Resource Quota Violation Audit

Threat/Impact Level: HIGH | Downtime Risk: HIGH | Time to Fix: 15–30 mins

TL;DR

• What broke: The namespace ResourceQuota has no remaining headroom (CPU, memory, pod count, or object count). The CronJob controller cannot CREATE the pod, so the job is silently dropped — no retry, no alert unless you're watching events.
• How to fix it: Either raise the quota on the namespace, reduce the CronJob's resource requests, or clean up completed/failed jobs consuming quota slots.
• Shortcut: Use our Client-Side Sandbox above to paste your CronJob YAML + kubectl describe quota output and auto-generate the refactored manifests.

Warning  FailedCreate  CronJob/cronjob-report-1712345678  
Error creating: pods "cronjob-report-1712345678-x9kzp" is forbidden: 
exceeds quota: namespace-quota, requested: requests.cpu=500m, 
used: requests.cpu=9800m, limited: requests.cpu=10000m

How to Fix It

Step 1 — Diagnose the Exact Quota Dimension

# See current usage vs hard limits
kubectl describe quota -n <namespace>

# See which pods are consuming the most requests
kubectl get pods -n <namespace> -o json | \
  jq '.items[] | {name: .metadata.name, cpu: .spec.containers[].resources.requests.cpu, mem: .spec.containers[].resources.requests.memory}'

# See zombie completed jobs eating quota
kubectl get pods -n <namespace> --field-selector=status.phase=Succeeded
kubectl get pods -n <namespace> --field-selector=status.phase=Failed

Basic Fix — Clean Up + Reduce Requests

# Delete completed job pods immediately
kubectl delete jobs --field-selector status.successful=1 -n <namespace>

# Patch the ResourceQuota if you own the namespace
kubectl patch resourcequota namespace-quota -n <namespace> \
  --patch '{"spec":{"hard":{"requests.cpu":"20000m"}}}'

Enterprise Best Practice — Fix the CronJob Manifest

The root cause is almost always missing or oversized resource requests. Right-size them using VPA recommendations or kubectl top pods.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: cronjob-report
  namespace: production
spec:
  schedule: "0 2 * * *"
+  startingDeadlineSeconds: 300
+  concurrencyPolicy: Forbid
+  successfulJobsHistoryLimit: 1
+  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
+      ttlSecondsAfterFinished: 600
      template:
        spec:
          containers:
          - name: report-runner
            image: company/report:1.4.2
            resources:
-             requests:
-               cpu: "500m"
-               memory: "512Mi"
-             limits:
-               cpu: "2000m"
-               memory: "2Gi"
+             requests:
+               cpu: "100m"
+               memory: "128Mi"
+             limits:
+               cpu: "500m"
+               memory: "256Mi"

Why each change matters:

• ttlSecondsAfterFinished: 600 — Garbage collects finished Job pods after 10 minutes, returning quota headroom automatically.
• successfulJobsHistoryLimit: 1 / failedJobsHistoryLimit: 1 — Prevents accumulation of dead Job objects.
• concurrencyPolicy: Forbid — Prevents overlapping runs from double-consuming quota.
• startingDeadlineSeconds: 300 — Limits the window for late starts; prevents the 100-missed-schedule lockout.
• Right-sized requests — The admission controller checks requests, not limits. Oversized requests are the #1 cause of premature quota exhaustion.

Enterprise Best Practice — Namespace Quota with Burst Headroom

apiVersion: v1
kind: ResourceQuota
metadata:
  name: namespace-quota
  namespace: production
spec:
  hard:
-   requests.cpu: "10000m"
-   requests.memory: "20Gi"
-   pods: "20"
+   requests.cpu: "20000m"
+   requests.memory: "40Gi"
+   pods: "50"
+   count/jobs.batch: "10"
+   count/cronjobs.batch: "20"

Pair this with a LimitRange to enforce default requests on any container that omits them:

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: production
spec:
  limits:
  - type: Container
    default:
      cpu: "200m"
      memory: "256Mi"
    defaultRequest:
      cpu: "50m"
      memory: "64Mi"

💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.

(For CI/CD pipeline automation, install via npm i -g stackengine).

# policy/cronjob_resources.rego
package main

deny[msg] {
  input.kind == "CronJob"
  container := input.spec.jobTemplate.spec.template.spec.containers[_]
  not container.resources.requests.cpu
  msg := sprintf("CronJob '%v': container '%v' is missing resources.requests.cpu", [input.metadata.name, container.name])
}

deny[msg] {
  input.kind == "CronJob"
  not input.spec.ttlSecondsAfterFinished
  msg := sprintf("CronJob '%v': ttlSecondsAfterFinished must be set to prevent quota leakage", [input.metadata.name])
}

# In your CI pipeline
conftest test cronjob.yaml --policy policy/

checkov -f cronjob.yaml --check CKV_K8S_11,CKV_K8S_12,CKV_K8S_13
# CKV_K8S_11: CPU requests set
# CKV_K8S_12: Memory requests set  
# CKV_K8S_13: CPU limits set

- alert: NamespaceQuotaCPUUsageHigh
  expr: |
    kube_resourcequota{resource="requests.cpu", type="used"} /
    kube_resourcequota{resource="requests.cpu", type="hard"} > 0.85
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Namespace {{ $labels.namespace }} CPU quota above 85%"
    description: "CronJobs will start failing at 100%. Used: {{ $value | humanizePercentage }}"

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-job-ttl
spec:
  validationFailureAction: enforce
  rules:
  - name: check-ttl
    match:
      resources:
        kinds: [CronJob]
    validate:
      message: "CronJob must set spec.jobTemplate.spec.ttlSecondsAfterFinished"
      pattern:
        spec:
          jobTemplate:
            spec:
              ttlSecondsAfterFinished: "?*"