Fixing 'error: unable to upgrade connection: container not found' in kubectl exec on Multi-Container Pods

Threat/Impact Level: MEDIUM | Downtime Risk: HIGH | Time to Fix: 2–5 mins

---

TL;DR

• What broke: kubectl exec on a multi-container pod defaults to no container when the pod has more than one container, and the API server returns container not found because the target is ambiguous or explicitly wrong.
• How to fix it: Add -c <container-name> to your kubectl exec command matching an exact name from spec.containers[*].name.
• Sandbox: Use our Client-Side Sandbox below to paste your pod spec and auto-generate the corrected exec command with all valid container targets listed.

---

The Incident (What Does the Error Mean?)

You ran something like:

kubectl exec -it my-pod -- /bin/sh

And the API server returned:

error: unable to upgrade connection: container not found ("")

or

error: unable to upgrade connection: container not found ("app")

Immediate consequence: You cannot exec into the pod. If this is a production incident and you need to inspect a running process, dump env vars, or run a diagnostic binary, you are blocked. Every minute counts.

The Kubernetes API server's /exec subresource requires an unambiguous container target. When a pod has two or more containers (sidecars, log shippers, service mesh proxies like Envoy, init containers, etc.), the API will not guess. If you omit -c or supply a name that doesn't match spec.containers[*].name exactly (case-sensitive), the request is rejected at the kubelet layer after the WebSocket upgrade attempt — hence the misleading "unable to upgrade connection" phrasing, which makes engineers chase network issues instead of the real cause.

---

The Attack Vector / Blast Radius

This is primarily an operational availability failure, but it has a secondary security surface:

Operational blast radius:

• During a live incident, engineers waste 10–20 minutes chasing kubelet logs, CNI issues, and RBAC policies before realizing the container name is wrong or missing.
• Automated runbooks and break-glass scripts that hardcode a single container name silently break the moment a sidecar (Istio, Datadog agent, Vault injector) is injected into the pod by a mutating webhook — without any change to your script. The injection happens transparently at admission time.
• CI/CD smoke tests using kubectl exec for health validation fail with cryptic errors, blocking deployments.

Security surface:

• If your RBAC pods/exec verb is scoped to a role without container-level granularity, an engineer trying to exec into the app container might accidentally target a privileged sidecar (e.g., a Vault agent with mounted secrets or a Falco sensor) if they guess the wrong name. Least-privilege exec should be paired with correct targeting.
• Misconfigured admission webhooks that inject containers with names colliding with your app container name will cause this error and can mask injection failures.

---

How to Fix It (The Solution)

Step 1: Enumerate All Containers in the Pod

kubectl get pod my-pod -o jsonpath='{.spec.containers[*].name}'
# Also check init containers:
kubectl get pod my-pod -o jsonpath='{.spec.initContainers[*].name}'

Example output: app envoy-proxy datadog-agent

Basic Fix

Add -c with the exact container name:

- kubectl exec -it my-pod -- /bin/sh
+ kubectl exec -it my-pod -c app -- /bin/sh

If the container name in the error message is wrong (e.g., you typed app but the actual name is application):

- kubectl exec -it my-pod -c app -- /bin/bash
+ kubectl exec -it my-pod -c application -- /bin/bash

Enterprise Best Practice

Harden your exec scripts and runbooks to dynamically resolve the primary container name rather than hardcoding it:

- CONTAINER="app"
- kubectl exec -it "$POD_NAME" -c "$CONTAINER" -- /bin/sh
+
+ # Resolve the first non-sidecar container dynamically from pod labels
+ CONTAINER=$(kubectl get pod "$POD_NAME" \
+   -o jsonpath='{.spec.containers[0].name}')
+ echo "Targeting container: $CONTAINER"
+ kubectl exec -it "$POD_NAME" -c "$CONTAINER" -- /bin/sh

For service mesh environments (Istio/Linkerd), the injected sidecar is typically appended — containers[0] is usually your app container. Validate this against your mutating webhook ordering.

RBAC least-privilege for exec (Kubernetes 1.26+ with container-level granularity via admission policies):

 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
   name: exec-role
 rules:
 - apiGroups: [""]
   resources: ["pods/exec"]
   verbs: ["create"]
+  resourceNames: ["my-pod"]  # Scope to specific pod names where possible

Pair with an OPA/Gatekeeper policy that validates the container field in the PodExecOptions admission request.

---

💡 Tired of pasting proprietary configs into ChatGPT? Generic AI tools log your company's ARNs, DB strings, and private keys. StackEngine is a zero-backend, pure Client-Side WASM utility. Drop your failing config into the sandbox above. We redact your secrets locally in the browser and auto-generate the refactored code using your own API key.

---

Prevention in CI/CD

1. OPA/Gatekeeper — Enforce Explicit Container Names in Exec

Write a Gatekeeper ConstraintTemplate that rejects pods/exec admission requests where the containerName field is empty. This forces all exec calls — including those from CI pipelines — to be explicit.

2. Checkov / Static Analysis on Runbook Scripts

Add a shell linter step in CI that scans for kubectl exec invocations missing -c:

# In your CI pipeline (GitHub Actions, GitLab CI, etc.)
grep -rn 'kubectl exec' ./scripts/ | grep -v '\-c ' && \
  echo "ERROR: kubectl exec found without -c flag" && exit 1

3. Kyverno Policy — Admission-Time Container Name Validation

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-exec-container-name
spec:
  rules:
  - name: check-exec-container
    match:
      resources:
        kinds: ["PodExecOptions"]
    validate:
      message: "kubectl exec must specify a container name explicitly via -c."
      pattern:
        container: "?*"

4. Mutating Webhook Awareness

Document which webhooks inject sidecars into your namespaces. After any Istio/Linkerd/Vault injection policy change, run a pod audit:

kubectl get pods -n production -o json | \
  jq '.items[] | {name: .metadata.name, containers: [.spec.containers[].name]}'

Pipe this into your CMDB or alert if container count deviates from the expected baseline.