Argo Rollouts Canary Analysis Metric Fix

Threat/Impact Level: HIGH | Downtime Risk: HIGH | Time to Fix: 15–30 mins

TL;DR

• What broke: Argo Rollouts AnalysisRun hit Failed phase because the metric query returned no data, the successCondition expression evaluated false, or the Prometheus/Datadog provider was misconfigured — rollout is now permanently blocked or auto-aborted.
• How to fix it: Correct the successCondition/failureCondition CEL expressions, fix the metric provider endpoint or query, and validate with kubectl argo rollouts get rollout <name> --watch.
• Shortcut: Use our Client-Side Sandbox above to auto-refactor your failing AnalysisTemplate YAML without leaking your internal metric endpoints.

Message: AnalysisRun default/myapp-canary-analysis-run-abc123 Failed
Reason:  Metric "error-rate" assessed Failed due to failureCondition 'result > 0.05' evaluated true
Status:  Phase: Failed
         Message: metric "success-rate" completed. Phase: Failed

Message: metric "p99-latency" assessed Inconclusive
Reason:  Insufficient data: no data returned from provider
Phase:   Inconclusive -> (after failureLimit exceeded) -> Failed

How to Fix It (The Solution)

Root Cause Checklist

Run this first:

# Get the full failure reason
kubectl describe analysisrun <run-name> -n <namespace>

# Check controller logs for provider errors
kubectl logs -n argo-rollouts deploy/argo-rollouts | grep -i "analysis\|metric\|error" | tail -50

# Manually test your Prometheus query
curl -G 'http://prometheus:9090/api/v1/query' \
  --data-urlencode 'query=sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))'

Basic Fix — Correcting `successCondition` and Query Scalar Return

The most common failure: Prometheus query returns a vector, but the condition expects a scalar.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate-analysis
spec:
  metrics:
  - name: error-rate
    interval: 1m
    failureLimit: 3
    provider:
      prometheus:
        address: http://prometheus.monitoring.svc.cluster.local:9090
        query: |
-         sum(rate(http_requests_total{status=~"5..",app="myapp"}[5m]))
-         /
-         sum(rate(http_requests_total{app="myapp"}[5m]))
+         scalar(
+           sum(rate(http_requests_total{status=~"5..",app="myapp"}[5m]))
+           /
+           sum(rate(http_requests_total{app="myapp"}[5m]))
+         )
-   successCondition: result[0] < 0.05
+   successCondition: result < 0.05
-   failureCondition: result[0] >= 0.05
+   failureCondition: result >= 0.05

Enterprise Best Practice — Full Hardened AnalysisTemplate

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: canary-success-rate
  namespace: production
spec:
+ args:
+ - name: service-name
+ - name: namespace
  metrics:
  - name: success-rate
-   interval: 30s
-   count: 5
+   interval: 1m
+   count: 10
+   initialDelay: 2m   # wait for metrics to populate post-deploy
    failureLimit: 2
+   inconclusiveLimit: 3  # prevent infinite Inconclusive -> stall
    provider:
      prometheus:
-       address: http://prometheus:9090
+       address: http://prometheus-operated.monitoring.svc.cluster.local:9090
        query: |
+         scalar(
            sum(rate(
-             http_requests_total{status!~"5.."}[2m]
+             http_requests_total{status!~"5..",service="{{args.service-name}}"}[5m]
            ))
            /
            sum(rate(
-             http_requests_total[2m]
+             http_requests_total{service="{{args.service-name}}"}[5m]
            ))
+         )
-   successCondition: result[0] >= 0.95
+   successCondition: result >= 0.99
-   failureCondition: result[0] < 0.95
+   failureCondition: result < 0.95

Critical additions explained:

• initialDelay: 2m — canary pods need time to receive traffic before metrics exist. Without this, the first scrape returns no data → Inconclusive → failure.
• inconclusiveLimit: 3 — caps how many no-data responses are tolerated before hard failure.
• scalar() wrapper — forces Prometheus to return a single float, not a labeled vector.
• Parameterized args — reuse the same template across services without hardcoding label selectors.

💡 Tired of pasting proprietary infrastructure configs into public LLMs? Generic AI interfaces log your company's ARNs, connection strings, and private keys. StackEngine enforces client-side zero-trust boundaries—the sandbox above executes 100% locally in your browser. Drop your broken manifest above to redact secrets locally and synthesize deterministic patch drafts with your own API key.

(For CI/CD pipeline automation, install via npm i -g stackengine).

kubectl apply --dry-run=server -f analysis-template.yaml

# policy/analysis_template.rego
package argorollouts

deny[msg] {
  input.kind == "AnalysisTemplate"
  metric := input.spec.metrics[_]
  not metric.initialDelay
  msg := sprintf("Metric '%v' is missing initialDelay — will fail on cold start", [metric.name])
}

deny[msg] {
  input.kind == "AnalysisTemplate"
  metric := input.spec.metrics[_]
  not metric.inconclusiveLimit
  msg := sprintf("Metric '%v' has no inconclusiveLimit — rollout can stall indefinitely", [metric.name])
}

# In your CI pipeline
conftest test analysis-template.yaml --policy policy/

# Validate query returns a scalar before it ever hits production
RESULT=$(curl -sf -G "$PROM_URL/api/v1/query" \
  --data-urlencode "query=$METRIC_QUERY" | jq '.data.result | length')

[ "$RESULT" -eq 1 ] || { echo "ERROR: Query returns vector or no data"; exit 1; }

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: argo-rollouts-analysis
rules:
- apiGroups: ["argoproj.io"]
  resources: ["analysisruns", "analysistemplates", "clusteranalysistemplates"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
  resources: ["secrets"]  # for metric provider tokens
  verbs: ["get"]