debug

Debug a running LlamaCloud release

Goal: take "something is broken" and produce a structured diagnosis: which pod, what symptom, which values.yaml field (or external dependency) is the root cause, and what to change. Read-only against the cluster.

For known error signatures, the catalog is in patterns.md next to this file. Read it lazily — only when you've narrowed down to a candidate symptom — to keep this skill's context budget small.

Step 0 — Confirm context

kubectl config current-context
helm list -A | grep -i llama

Read the context back to the user and confirm. Identify the release(s). If multiple, ask which one.

Step 1 — Health snapshot

helm status <release> -n <ns>
helm history <release> -n <ns> | tail -10
kubectl get pods -n <ns> -o wide
kubectl get events -n <ns> --sort-by=.lastTimestamp | tail -50
kubectl get pdb,hpa,ingress,svc,pvc -n <ns>

Tabulate the pod list. Note the deployment naming convention: the LlamaCloud components have fixed deployment names (the chart does not prefix them with the release name), while subchart components do use the <release>- prefix.

Group	Deployment / pod prefix	What's normal
Backend	llamacloud-* (yes — bare release-named, not llamacloud-backend)	1+ Ready
Frontend	llamacloud-web-*	1+ Ready
Jobs service	llamacloud-operator-*	1+ Ready
Jobs worker	llamacloud-worker-*	1+ Ready
Usage / telemetry	llamacloud-telemetry-*	1 Ready
Parse	llamacloud-parse-*	1+ Ready
Parse OCR (GPU)	llamacloud-ocr-*	0+ Ready depending on config.parseOcr.enabled
Layout detection (GPU)	llamacloud-layout-*	0+ depending on config.parseLayoutDetection.enabled
Layout V3 (GPU)	llamacloud-layout-v3-*	0+ depending on config.parseLayoutDetectionV3.enabled
Parse Temporal worker	llamacloud-temporal-parse-*	1+ Ready when temporal workloads enabled
Jobs Temporal worker	temporal-jobs-worker-*	1+ Ready when temporal workloads enabled
Temporal subchart	<release>-temporal-subchart-*	frontend / history / matching / worker / web / admintools all Ready
llama-agents subchart	varies	optional

When the user gives a pod name, treat the bare llamacloud-<hash>-<random> as the backend unambiguously. This trips up newcomers because the obvious guess is "frontend".

To unambiguously isolate one component, use label selectors instead of name globs:

kubectl get pods -n <ns> -l app.kubernetes.io/name=llamacloud           # backend only
kubectl get pods -n <ns> -l app.kubernetes.io/name=llamacloud-web        # frontend
kubectl get pods -n <ns> -l app.kubernetes.io/name=llamacloud-worker     # jobs worker
kubectl get pods -n <ns> -l app.kubernetes.io/name=llamacloud-parse      # parse
# ...etc — every component has matching app.kubernetes.io/name

When <component>.horizontalPodAutoscalerSpec is set on a component (default: empty, no HPA), the chart creates a standard HorizontalPodAutoscaler named after the Deployment (e.g. llamacloud, llamacloud-web). If you see HPAs with different naming (e.g. keda-hpa-*-scaledobject), that's the customer's KEDA / external autoscaler managing the same Deployments — the Deployment/<name> reference is authoritative regardless of how the HPA was named.

Standard Secrets the chart creates (look for these when debugging missing-secret errors). Each is generated only when the corresponding values block is configured inline; if the user sets the matching *.secret: reference instead, the chart skips generating its own Secret and reads from the user-provided one.

Always created (when the corresponding values block is set inline):

Secret	Triggered by	Purpose
llamacloud-license-key	license.key	License
postgresql-secret	postgresql.password	Postgres credentials
mongodb-secret	mongodb.password or mongodb.mongodb_url	MongoDB credentials / connection URL
rabbitmq-secret	rabbitmq.password or rabbitmq.connectionString	RabbitMQ credentials
redis-secret	redis.password	Redis credentials
bucket-secret	config.storageBuckets.* credentials	Object storage credentials (for non-IRSA setups, e.g. static AWS keys, GCS HMAC, Azure SAS)

Conditional — only when the matching feature is enabled:

Secret	Triggered by	Purpose
oidc-secret	config.authentication.oidc.enabled: true	OIDC client credentials
basic-auth-secret	basic auth enabled (mutually exclusive with OIDC)	Basic auth credentials
temporal-postgresql-secret	temporal.deploy: true	Separate Postgres credential copy for the Temporal subchart
qdrant-secret	qdrant.enabled: true	Qdrant API key
s3proxy-secret	config.storageBuckets.s3proxy.enabled: true	s3proxy upstream credentials (e.g. GCS HMAC, Azure storage key)

LLM provider Secrets — one per configured provider:

Secret	Triggered by	Purpose
openai-api-key-secret	config.llms.openAi.apiKey set	OpenAI API key
anthropic-api-key-secret	config.llms.anthropic.apiKey set	Anthropic API key
gemini-api-key-secret	config.llms.gemini.apiKey set	Google Gemini API key
azure-open-ai-api-key-secret	config.llms.azureOpenAi.deployments[] configured	Azure OpenAI deployment keys (one Secret holding keys for all configured deployments)
aws-bedrock-api-key-secret	config.llms.awsBedrock block present	AWS Bedrock config (model versions; auth via IRSA, not a static key, but the Secret still carries config values)
google-vertex-ai-api-key-secret	config.llms.googleVertexAi.credentialsJson set	Vertex AI service-account JSON
llm-provider-configs-secret	always (when any LLM provider configured)	Aggregated provider config — base URLs, model names, deployment names, etc. — referenced by backend env vars

Ingress TLS (NOT created by the chart):

Secret	Triggered by	Purpose
<ingress.tlsSecretName>	ingress.enabled: true references it	Ingress TLS. The chart does not create this Secret; the user provides it (cert-manager, externally-managed cert, or kubectl create secret tls).

ConfigMaps the chart creates:

Centralized config (always present when chart deployed):

ConfigMap	Purpose
common-config	Shared env shared across most deployments
urls-config	Intra-cluster Service URLs each component talks to
bucket-config	Object storage bucket names and provider
feature-config	Feature flags
extract-config	Extract-specific tuning
concurrency-config	Concurrency limits per workload
rate-limits-config	Rate limits per LLM / API
temporal-connection-config	Temporal frontend host/port

Conditional:

ConfigMap	Triggered by	Purpose
llama-agents-config	llamaAgents.deploy: true or llamaAgents.controlPlaneUrl set	llama-agents wiring
s3proxy-config	config.storageBuckets.s3proxy.enabled: true	s3proxy JCLOUDS configuration (provider, region, endpoint)

Per-deployment ConfigMaps (one per component, named after the Deployment):

llamacloud-web, llamacloud-worker, llamacloud-parse, llamacloud-temporal-parse, temporal-jobs-worker, etc. — these hold component-specific env. When debugging a single component's environment, this is the ConfigMap to read.

Temporal subchart ConfigMaps (when temporal.deploy: true):

<release>-temporal-subchart-dockerize-config, <release>-temporal-subchart-dynamic-config — Temporal server configuration.

For each pod that is not Ready or has restarted, capture the snapshot row.

Step 2 — Symptom routing

For each unhealthy pod, classify by status.phase + containerStatuses[].state + last event:

Symptom	First action	Then
Pending, FailedScheduling	kubectl describe pod → read Events:	check node capacity, taints, GPU availability
Pending, Unschedulable due to PVC	kubectl get pvc -n <ns> and kubectl get pv	StorageClass / PV provisioner issue
Waiting: ImagePullBackOff / ErrImagePull	kubectl describe pod → read pull error	imagePullSecrets, image tag existence, registry reachability
Waiting: CreateContainerConfigError	kubectl describe pod → secret/configmap reference	named secret or key missing; check <release>-*-env Secrets
Waiting: CrashLoopBackOff	kubectl logs <pod> --previous	match against patterns.md
Running, 0/1 Ready, restartCount stable	readiness probe failing	kubectl describe pod → readiness probe + check Service endpoints
Running, 0/1 Ready, restartCount climbing	liveness probe killing it	kubectl describe pod → liveness probe + recent logs
Terminated: OOMKilled	check resources.limits.memory vs kubectl top pod	bump limit or downsize work
Terminated: Error, exit code 137	similar to OOM (signal 9)	check node memory pressure
Terminated: Error, exit code 143	SIGTERM during graceful shutdown — usually rollout, not a bug	only worry if it's an active loop

For pods in CrashLoopBackOff, always grab:

kubectl logs <pod> -n <ns> --all-containers --previous --tail=200
kubectl logs <pod> -n <ns> --all-containers --tail=100
kubectl describe pod <pod> -n <ns>

Then read the first 5–10 ERROR/FATAL lines and look them up in patterns.md.

Step 3 — Cross-pod / network checks

Some failures aren't a single bad pod, they're a routing problem between pods. Common ones:

# Service endpoints — if empty, no Pods match the Service selector or none are Ready
kubectl get endpoints -n <ns>

# DNS from inside a healthy pod (if any exist)
# Pick a Running pod; nslookup of dependency hostnames and sibling Services.
# NOTE: the LlamaCloud backend Service is named `llamacloud` (not
# `<release>-backend`); see the deployment-naming table above. Subchart
# Services like temporal DO use the `<release>-` prefix.
kubectl exec -n <ns> <healthy-pod> -- nslookup llamacloud
kubectl exec -n <ns> <healthy-pod> -- nslookup llamacloud-web
kubectl exec -n <ns> <healthy-pod> -- nslookup <release>-temporal-subchart-frontend
# (kubectl exec is read-only-equivalent for diagnosis; ask user before running this)

# NetworkPolicy egress
kubectl get networkpolicy -A
# The chart itself ships only ONE NetworkPolicy (`<release>-agent-backend-egress`,
# in the namespace configured by `llama-agents-subchart.apps.namespace` (default
# `llama-agents`), only when llamaAgents.deploy is true and
# llamaAgents.allowBackendEgress is true). It allows egress from agent pods TO
# the backend, not the other way around. Any other NetworkPolicy you see is
# customer-managed and may be blocking pod-to-pod traffic.

If kubectl exec is sensitive in the customer's environment (it usually is in production), ask before running and explain what you're checking. If declined, continue with describe/logs only.

Step 4 — Cloud-side checks (only if relevant)

If the symptom points at a managed dependency (DB connection refused, S3 access denied, OIDC discovery failing), and the cluster is on AWS/Azure/GCP, run the same read-only cloud checks the preinstall-check skill uses (Phase 4). Specifically:

• DB unreachable → aws ec2 describe-security-groups cross-check between cluster SG and DB SG
• S3 access denied → aws iam get-role --role-name <irsa-role> and check the bucket policy / IRSA trust relationship
• OIDC failures → curl <discoveryUrl> from the user's laptop and from a debug pod (with consent)

Ask before each cloud CLI call. Read-only by definition.

Step 5 — Match against patterns.md

patterns.md (next to this file) catalogs the highest-frequency LlamaCloud-specific failure signatures and maps each to:

• The error message you'll see in logs
• The likely root cause
• The values.yaml field to change (or external action to take)
• A pointer to the relevant examples/*.yaml

Read it now if any captured log line matches a pattern entry. Only read the relevant section, not the whole file — patterns.md is organized by component to make this efficient.

Step 6 — Write debug-report.md

Always write the report. This is what the user sends to support if you can't fix it.

# LlamaCloud debug report — <ISO timestamp UTC>

## Summary
<one-sentence statement of what's wrong, e.g. "3 backend pods in CrashLoopBackOff, all failing on Postgres connection refused. Most likely cause: cluster SG missing from RDS inbound rules.">

## Inputs
- Release: <name>
- Namespace: <ns>
- Chart version: <`helm get metadata <release>` output>
- Cluster context: <ctx>
- Cluster cloud: <aws|azure|gcp|other>

## Pod state
<table of all pods with phase / ready / restarts / age>

## Failing pods — evidence
<for each failing pod>:
  ### <pod-name>
  Symptom: <one-line>
  Container state: <Waiting/CrashLoopBackOff/etc>
  Recent events: <bulleted, sorted by time>
  Last log lines (previous container): <fenced code block, redacted>
  Matched pattern: <patterns.md anchor or "no match">
  Suggested fix: <concrete>

## Recent helm history
<helm history output>

## Cluster events (last hour)
<filtered event list>

## Cloud routing checks
<if run; what was confirmed / what was off>

## Recommended next steps
1. <highest-priority action>
2. <second-priority action>
3. ...

## What to send to support
This report (after a final secret review). Specifically:
- The above
- The output of `kubectl logs <pod> --previous --tail=500` for each failing pod (already captured above; re-run if more context needed)
- `helm get values <release> -n <ns> --revision <rev>` with secrets masked

Mask all secrets before saving. Run a final regex sweep over the report content for known secret patterns: password, apiKey, clientSecret, key, token, connectionString, credentialsJson, JWT-shaped strings, AWS access key ids, RSA/EC PEM headers. If unsure, mask.

Critical rules

• Read-only. No kubectl edit, kubectl patch, kubectl delete, kubectl rollout restart, helm upgrade, helm rollback. Diagnose and recommend; do not "just try restarting the pod."
• Ask before kubectl exec. It's technically read-only, but customers are sensitive about it. Explain what you'd check and let them say no.
• Mask secrets in every file you write. Run a regex sweep before declaring a report written.
• Don't re-run preinstall checks aggressively. If the install is up but unhealthy, the relevant subset of preinstall checks is fine — but full Phase 4 cloud surveys can flood the customer's CloudTrail with surprising activity.
• One symptom at a time. If multiple pods are unhealthy with different errors, work through them in priority order (backend > workers > optional services), not in parallel — they often have a shared root cause.
• Stop on cancel. If the user says "stop" or "that's enough", stop and write whatever is in the report so far.