Fixing ArgoCD 'sync failed: error: unknown' SSH Key Errors in GitOps Pipelines

Threat/Impact Level: HIGH | Downtime Risk: HIGH | Time to Fix: 10–20 mins

TL;DR

• What broke: ArgoCD's argocd-repo-server cannot authenticate to the Git remote over SSH — the deploy key is missing, has wrong permissions, uses an unsupported algorithm, or the known_hosts entry is absent, causing all syncs to fail with the opaque error: unknown.
• How to fix it: Re-register the SSH key as a valid Kubernetes Secret, ensure known_hosts is populated via argocd cert add-ssh, and verify the key algorithm is ed25519 or rsa (4096-bit minimum).
• Shortcut: Use our Client-Side Sandbox below to auto-refactor your ArgoCD repo secret YAML — it redacts private key material locally before analysis.

---

The Incident (What Does the Error Mean?)

Raw log output from argocd-repo-server:

time="2024-05-10T03:17:42Z" level=error msg="git fetch origin" error="unknown"
ERROR[0012] error: unknown
Failed to sync app 'my-app': error: unknown
rpc error: code = Unknown desc = error: unknown

This is ArgoCD swallowing the underlying git subprocess error. The actual failure is one of:

• SSH key not found or not mounted into argocd-repo-server
• known_hosts missing the remote host fingerprint (strict host key checking blocks the connection)
• Private key algorithm rejected by the Git server (e.g., DSA, weak RSA)
• Secret key field name mismatch (sshPrivateKey vs ssh-privatekey)

Immediate consequence: Every application tied to this repository is stuck. No rollouts, no hotfixes, no rollbacks. Your GitOps pipeline is completely dead.

---

The Attack Vector / Blast Radius

This is not just an outage — it is a security configuration failure with real exploit surface:

1. Exposed private keys in plaintext ConfigMaps or Pod env vars. Engineers under pressure during an outage frequently dump keys into the wrong Kubernetes resource type, making them readable by any pod with default RBAC.
2. Overly permissive deploy keys. If the SSH key has write access to the repo (common mistake), a compromised argocd-repo-server pod means an attacker can push malicious manifests directly to your GitOps source of truth.
3. StrictHostKeyChecking=no workarounds. The fastest wrong fix. Disabling host key checking opens the connection to MITM attacks — an attacker on the network path can intercept and replace your manifests in transit.
4. Blast radius: A single broken repo credential blocks ALL ArgoCD applications pointing to that repo. In a monorepo setup, that is your entire fleet.

---

How to Fix It

Step 1 — Diagnose the actual SSH error

Exec into the repo server and test the connection manually:

kubectl exec -it -n argocd deploy/argocd-repo-server -- \
  ssh -i /app/config/reposerver/ssh/sshPrivateKey \
  -o StrictHostKeyChecking=yes \
  -T [email protected]

If you see Host key verification failed, your known_hosts is the problem. If you see Permission denied (publickey), the key is wrong or not registered on the Git server.

---

Basic Fix — Re-register the Repository Credential

Delete and re-add the repo via ArgoCD CLI:

argocd repo rm [email protected]:your-org/your-repo.git

argocd repo add [email protected]:your-org/your-repo.git \
  --ssh-private-key-path ~/.ssh/argocd_ed25519 \
  --insecure-ignore-host-key  # TEMPORARY ONLY — remove after fixing known_hosts

Add the correct known_hosts entry:

ssh-keyscan github.com | argocd cert add-ssh --batch

---

Enterprise Best Practice — Correct Secret Structure + Key Algorithm

The ArgoCD repo secret must use the exact field name sshPrivateKey under stringData. Wrong field names silently fail.

apiVersion: v1
kind: Secret
metadata:
  name: my-gitops-repo
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  type: git
  url: [email protected]:your-org/your-repo.git
- ssh-privatekey: |
-   -----BEGIN RSA PRIVATE KEY-----
-   MIIEowIBAAKCAQEA1234...  # weak 1024-bit RSA, wrong field name
-   -----END RSA PRIVATE KEY-----
+ sshPrivateKey: |
+   -----BEGIN OPENSSH PRIVATE KEY-----
+   b3BlbnNzaC1rZXktdjEAAAAA...  # ed25519 key, correct field name
+   -----END OPENSSH PRIVATE KEY-----

Generate a correct ed25519 deploy key:

# Generate — no passphrase for automated use
ssh-keygen -t ed25519 -C "argocd-deploy@your-org" -f ./argocd_ed25519 -N ""

# Register public key as read-only deploy key on GitHub/GitLab
cat argocd_ed25519.pub  # paste into repo Settings > Deploy Keys (READ ONLY)

# Create the Kubernetes secret
kubectl create secret generic my-gitops-repo \
  --from-file=sshPrivateKey=./argocd_ed25519 \
  --dry-run=client -o yaml | kubectl apply -f -

# Label it so ArgoCD picks it up
kubectl label secret my-gitops-repo \
  -n argocd argocd.argoproj.io/secret-type=repository

Verify ArgoCD sees the repo as connected:

argocd repo list
# STATUS column must show: Successful

---

💡 Tired of pasting proprietary configs into ChatGPT? Generic AI tools log your company's ARNs, DB strings, and private keys. StackEngine is a zero-backend, pure Client-Side WASM utility. Drop your failing config into the sandbox above. We redact your secrets locally in the browser and auto-generate the refactored code using your own API key.

---

Prevention in CI/CD

1. Enforce Secret Structure with Conftest/OPA

Add a policy that fails the pipeline if an ArgoCD repo secret is missing the correct label or uses the wrong field name:

# policy/argocd_repo_secret.rego
package argocd

deny[msg] {
  input.kind == "Secret"
  input.metadata.labels["argocd.argoproj.io/secret-type"] == "repository"
  not input.stringData.sshPrivateKey
  msg := "ArgoCD repo secret must use 'sshPrivateKey' field, not 'ssh-privatekey'"
}

deny[msg] {
  input.kind == "Secret"
  input.metadata.labels["argocd.argoproj.io/secret-type"] == "repository"
  contains(input.stringData.sshPrivateKey, "BEGIN RSA PRIVATE KEY")
  msg := "Legacy PEM RSA keys are forbidden. Use ed25519 (OPENSSH format)."
}

2. Checkov scan on Kubernetes manifests

checkov -d ./k8s/argocd --framework kubernetes \
  --check CKV_K8S_35  # Secrets should not be hardcoded in env vars

3. Rotate deploy keys via CI on a schedule

# .github/workflows/rotate-argocd-keys.yml
name: Rotate ArgoCD Deploy Keys
on:
  schedule:
    - cron: '0 2 1 * *'  # First of every month
jobs:
  rotate:
    runs-on: ubuntu-latest
    steps:
      - name: Generate new ed25519 key
        run: ssh-keygen -t ed25519 -f argocd_ed25519 -N ""
      - name: Update GitHub deploy key via API
        run: |
          # Delete old key, register new public key via GitHub API
          # Update Kubernetes secret via kubectl/Vault

4. Use External Secrets Operator or Vault

Never store the raw private key in a Kubernetes Secret long-term. Sync it from Vault:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: argocd-repo-ssh
  namespace: argocd
spec:
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: my-gitops-repo
    template:
      metadata:
        labels:
          argocd.argoproj.io/secret-type: repository
  data:
    - secretKey: sshPrivateKey
      remoteRef:
        key: secret/argocd/deploy-keys
        property: github_ed25519_private