agento11y-instrument

Agent Observability — instrument an LLM app

Help a developer wire their own LLM app or agent to Grafana Agent Observability, from zero or from a broken/partial state, and keep going until data actually lands in the stack. The value this skill adds over the static instrumentation prompt is two things a prompt can't do:

1. A mechanical gap checklist run against the real code — headed by the one failure that is completely silent (missing OTel providers → every metric lost, no error).
2. A verification loop through gcx: after each change, run the app and check the gcx agento11y agents / conversations / generations commands to confirm generations arrived. Diagnose the next gap from what's missing, not from guesswork.

The SDK reference (env vars, provider snippets, field lists, framework adapters, workflow steps) lives in agento11y's llms.txt "Path B". Fetch it rather than restating it here; this file holds the flow and the decision logic. A minimal fallback lives in references/instrumentation.md for when the fetch is unavailable.

Rules

• Reference, don't restate. Fetch SDK detail from https://raw.githubusercontent.com/grafana/agento11y/main/llms.txt (Path B). Only inline decision logic here. If the fetch fails, fall back to references/instrumentation.md.
• Never invent an endpoint or a token. Read them from the environment (AGENTO11Y_ENDPOINT, AGENTO11Y_PROTOCOL, AGENTO11Y_AUTH_MODE, AGENTO11Y_AUTH_TENANT_ID, AGENTO11Y_AUTH_TOKEN, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS) or ask the developer. Never fabricate a URL or mint a token.
• Target is Grafana Cloud. The developer supplies the endpoint + token (Step 0), and the gcx verification loop (Step 5) confirms data landing against the Cloud tenant. Never fabricate the endpoint or token — read them from the environment or ask.
• Write AGENTO11Y_* env vars, never SIGIL_*. SIGIL_* is a deprecated legacy fallback. Do this even if sibling apps or existing .env files in the repo use SIGIL_* — matching a stale local convention perpetuates it. If the app already reads SIGIL_*, add the AGENTO11Y_* names (the SDK still honors both) and note the old ones are deprecated. Do not "match the siblings."
• Gate every code WRITE on explicit confirmation. Report first (Step 4), apply only after the developer says yes (Step 5). Read-only gcx verification and re-running the app happen freely inside the loop; editing files does not.
• Keep diffs small; do not change app behavior. Instrumentation is additive. No refactors, no prompt rewrites, no dependency upgrades beyond the SDK/adapter packages actually needed.
• Never change the model, provider, or the app's LLM config — not even with permission, not even "just to run the verify loop." Instrument whatever model the app already uses. This is absolute: changing the model is out of scope for instrumentation, full stop. If a run fails because a provider API key is missing, the only allowed responses are: (a) ask the developer to provide the key for the model the app already uses, or (b) skip the live run and report the wiring as verified-by-construction, telling the developer to run it themselves. Do not offer to switch the provider, do not ask "which provider should I use?", and do not add a new provider dependency (e.g. langchain-anthropic) to make the run succeed. If the developer separately says they want a different model, that is an app change they own — tell them to make it and re-invoke this skill; do not fold it into the instrumentation diff. Swapping the model silently changes what the app does and what gets observed, which defeats the point. The provider API key (ANTHROPIC_API_KEY, OPENAI_API_KEY, …) is the app's own concern, not the instrumentation's — it authenticates the LLM call, not the telemetry export, and the app already has it if it runs at all. So don't ask for it, configure it, or rewire it; if the live verify-run fails on a missing provider key, skip the run and report verified-by-construction (see Step 5). Just don't conflate the two 401s: a 401 on generation ingest is observability auth and is yours to fix (usually a missing AGENTO11Y_PROTOCOL/AGENTO11Y_AUTH_MODE); an auth error from the model provider is not — surface it and let the developer handle their own key.
• Do not assume language symmetry. Verify the provider wrapper / framework adapter actually exists for the app's language before recommending it (Python has the most adapters, JS fewer, Go only google-adk, Java/.NET core + providers + google-adk). If it doesn't exist, hand-instrument with the core SDK. Prefer, in order: provider wrapper → framework adapter → hand-instrumentation.
• The loop is bounded. At most ~3–4 instrument→verify iterations. If data still isn't landing, stop and report what's checked and what remains — don't loop forever.
• Field-name traps: cache_write_input_tokens, NOT cache_creation_input_tokens. agent_version maps to the gen_ai.agent.version label and is required for per-version Performance charts. MessageRole (Python SDK) has only USER, ASSISTANT, TOOL — there is no SYSTEM (or DEVELOPER) member; MessageRole.SYSTEM raises AttributeError. Fold the system prompt into the USER message (or a text_part), and prefer the user_text_message() / assistant_text_message() / tool_result_message() helpers over hand-building Message(role=...). Always confirm enum members and helper names against the installed package before running — do not assume from llms.txt.
• Out of scope: offline test suites → agento11y-test-starter; tenant eval rules + guards on real traffic → agento11y-prod-setup. Coding-agent telemetry plugins (Claude Code, Cursor, …) → llms.txt "Path A". Any control-plane write.
• If a required input is missing (entrypoint, framework, endpoint, gcx auth), ask — don't guess.

Step 0 — Credentials and endpoint

The app needs, in its environment before the SDK starts, seven vars — not five; the two mode vars are the ones most often forgotten, and their absence is a silent 401:

• generation ingest: AGENTO11Y_ENDPOINT, AGENTO11Y_PROTOCOL=http, AGENTO11Y_AUTH_MODE=basic, AGENTO11Y_AUTH_TENANT_ID, AGENTO11Y_AUTH_TOKEN.
• traces/metrics: OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS.

AGENTO11Y_PROTOCOL=http and AGENTO11Y_AUTH_MODE=basic are required for Cloud, not optional — the SDK defaults (grpc / no-auth) return a silent 401 against the Cloud HTTP ingest endpoint. (They scope the ingest channel only; the OTel channel's transport/auth is set entirely by the OTEL_* vars — see references/instrumentation.md.) The only var that is sometimes omittable is OTEL_EXPORTER_OTLP_HEADERS: required when sending directly to the Cloud OTLP gateway (the common case, gateway enforces Basic auth), omittable only when OTEL_EXPORTER_OTLP_ENDPOINT points at a local Alloy / OTel Collector that already holds the Cloud credentials. First check what's already set (including any existing .env) — if all are present, skip to Step 1. Watch for the near-miss where the endpoint is set under the wrong name (e.g. AGENTO11Y_API_ENDPOINT — the SDK reads AGENTO11Y_ENDPOINT, so the wrong name is silently ignored and ingest falls back to a default host).

When any value is missing, do NOT just list the variable names and ask — hand the developer the exact place to get each one (link + clicks), every time. The concrete sources are in point 3 below; surface them proactively. The most common failure of this skill is naming OTEL_EXPORTER_OTLP_ENDPOINT/OTEL_EXPORTER_OTLP_HEADERS and leaving the developer to guess — the answer is the stack OTLP tile + "Generate now", which precomputes both. Give that first.

What gcx does for you (run these):

1. gcx config current-context — is there a working context? If not, just ask the developer to log in to the stack they want the instrumentation to connect to — e.g. "run gcx login against your stack." Do not fabricate the login command yourself (don't guess the host or flags); let them run their own login (the Agent Observability setup screen gives them the exact command, or they use gcx login). Instrumentation itself needs no gcx login — only Step 5 verification does, so this never blocks writing the code.
2. gcx cloud orgs list — discover organisation slugs using browser Cloud OAuth. If re-authentication is needed, use gcx cloud login with the same context; default scopes include profile and stack management. Access-policy tokens cannot list memberships and override OAuth when GRAFANA_CLOUD_TOKEN or cloud.<entry>.token is set. With access-policy authentication, ask the developer for their organisation slug instead. Have the developer select the intended slug, then run gcx cloud stacks list --org <org-slug> and gcx cloud stacks get <stack-slug> to identify the target stack and its URLs. This gives you the stack to point the developer at, and confirms which tenant the Step 5 verification will read from.

What still needs the Connection page (gcx cannot do these today):

1. gcx does not generate the Agent Observability OTLP gateway URL, and does not mint the ingest / access-policy token. When you ask the developer for a value, always tell them exactly where to get it — a link and the clicks — never just name the variable and wait. The two channels come from two different places:

   OTEL_* (traces/metrics) — easiest, let Cloud build them. Send the developer to the stack's OTLP tile: https://grafana.com/orgs/<org-slug>/stacks/<stack-id>/otlp-info. It already shows OTEL_EXPORTER_OTLP_ENDPOINT and the Instance ID; under Password / API Token → "Generate now" it mints a token and then fills an Environment Variables block with all OTEL_* vars — the base64 OTEL_EXPORTER_OTLP_HEADERS is precomputed, ready to copy. No manual base64. (In Python, the value uses Basic%20… — keep it as given.)

   AGENTO11Y_* (generation ingest) — the plugin Connection page. AGENTO11Y_ENDPOINT and the token come from https://<stack>.grafana.net/plugins/grafana-agento11y-app → Connection tab. When the developer creates the token via "Create a token in Cloud Access Policies", tell them the scopes: sigil:write, metrics:write, traces:write, logs:write. UI heads-up: sigil is not in the default resource list — add it via "Add scope" (then tick Write); the scope is still sigil:* (the Cloud resource keeps the old name). The same glc_… token works for both channels if it has all four scopes. Also set AGENTO11Y_PROTOCOL=http and AGENTO11Y_AUTH_MODE=basic (see references/instrumentation.md — the SDK defaults grpc/none give a 401).

   Ask the developer to put the values in a gitignored .env (or export them) themselves — do not ask them to paste a secret token into the chat (it is captured in the transcript). Instrument the code to read from the environment and have them supply the values out-of-band. Never invent a URL or mint a token.

   Two different tokens — don't confuse them. gcx logs in with its own OAuth token (gat_) and refreshes it automatically; that is what authenticates the gcx commands here. It is not the app's ingest token. The app needs a separate access-policy token (glc_…) in AGENTO11Y_AUTH_TOKEN / OTEL_EXPORTER_OTLP_HEADERS, and gcx does not create that one.

Once gcx has a working context, the Step 5 verification commands (under the gcx agento11y group) work against the developer's tenant even before the app's own credentials are fully wired.

The Connection page is the only manual step. If a future gcx release can create the access-policy token and surface the OTLP endpoint, this step collapses to gcx-only — but do not assume it can today; check gcx cloud --help rather than promising it.

Step 1 — Read the app and detect language / framework / shape

Find and read, recording file:line for each:

1. The generation entrypoint(s) — where the model is invoked.
2. How the LLM client is constructed (which provider: OpenAI / Anthropic / Gemini / other).
3. The app bootstrap / init — where an OTel TracerProvider / MeterProvider would be created.
4. Any existing Agent Observability SDK imports (agento11y / @grafana/agento11y / the Go agento11y package — or the legacy sigil_sdk / @grafana/sigil-sdk-js / Go sigil names in older code) or AGENTO11Y_* usage.

Detect:

• Language — from the manifest / extensions (pyproject.toml/.py, package.json/.ts, go.mod/.go, gradle/.java, .csproj/.cs).
• Framework — grep for langgraph, langchain, openai-agents, llamaindex, google-adk, strands, pydantic-ai, litellm, claude-agent-sdk, vercel-ai-sdk, crewai, or a custom orchestrator.
• Shape — single generation vs agentic pipeline (multiple nodes / a graph / sub-agents). This decides whether workflow steps (checklist #8) and parent links (#7) are in play.

Before recommending a provider wrapper or framework adapter, confirm it exists for this language (fetch llms.txt "SDK API surface" — the matrix is not symmetric across languages).

Step 2 — Classify instrumentation state

State the classification explicitly, with the file:line evidence that led to it:

• none (greenfield) — no SDK import anywhere. Full setup from scratch.
• partial — an SDK client is constructed and some generations are wrapped, but coverage is incomplete (no OTel providers, no agent_version, no workflow steps for an agentic pipeline, no parent links). Run the full checklist; recommend + apply only the gaps.
• broken — the SDK is present but wrong: metrics silently lost (no MeterProvider), export misconfigured, legacy SIGIL_* vars, stream/non-stream mode mismatch, set_result/SetResult never called, or rec.err()/Err() never checked. Fix first, then gap-check.

All three paths converge on the same checklist (Step 3); they differ only in how much is already done.

Step 3 — Run the instrumentation gap checklist

Walk each item against the code. Record PRESENT / MISSING / WRONG with file:line. This mechanical audit is the skill's core value. Items 0, 1, 2, 5, 6 fail silently (no error) — always check them. Items 3, 7, 8 mean data lands but analysis is degraded. For the fix, read the named section of the fetched llms.txt (locate it by its heading — do not trust line numbers, they drift).

#	Check	Silent-failure symptom	llms.txt section
0	The .env actually takes effect. Confirm the app loads its own .env by an explicit path (not a bare load_dotenv() resolved by CWD) and that it wins over vars already in the environment. Verify by printing os.environ["AGENTO11Y_ENDPOINT"] / OTEL_EXPORTER_OTLP_ENDPOINT after all imports, not before	import litellm (and some other libs) inject localhost OTLP/ingest defaults into os.environ at import time (OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4318, AGENTO11Y_ENDPOINT=localhost:8080). A plain load_dotenv() does not override already-set vars → the Cloud endpoints in .env never apply and everything ships to localhost, returning 200 OK if a local stack is up. Zero error signal, and gcx against the Cloud tenant shows nothing. Fix: load_dotenv(<path-relative-to-__file__>, override=True) before constructing providers/client. A bare load_dotenv() also resolves the wrong .env by CWD	"Environment"
1	OTel TracerProvider and MeterProvider created before the SDK client (verify by construction + Performance view / OTLP POSTs — not via gcx, which can't see OTel; see Step 5)	spans/metrics go to no-op → all latency/token/cost metrics lost. The #1 failure.	"OTel setup (required)"
2	Providers shut down after shutdown()	last batch of spans/metrics dropped on exit	"OTel setup (required)"
3	agent_name + agent_version set on generations / handlers	per-version Performance charts break (join on gen_ai.agent.version)	"Agent Observability architecture and ingest model", "Telemetry fields to prioritize"
4	set_result/SetResult includes response_id, response_model, finish/stop reason, full token usage (incl. cache_read_input_tokens, cache_write_input_tokens, reasoning_tokens), and input/output populated with Message objects (system+user prompt in input, model reply in output)	charts/cost blank; wrong cache_creation_input_tokens name silently ignored; empty input/output → the conversation thread shows "No messages in this turn" — tokens land but there is no visible prompt/response	"Implementation rules", "Telemetry fields to prioritize"
5	rec.err()/Err() checked after the recorder closes	SDK validation/enqueue errors are silent → generations never arrive, no signal	"Implementation rules"
6	SYNC (non-stream) vs STREAM (stream) set correctly	streaming metrics (TTFT) corrupted	"Agent Observability architecture and ingest model", "Implementation rules"
6b	operation_name is a recognized value — generateText (SYNC default), streamText (STREAM default), embeddings, execute_tool, framework_chain, framework_retriever. Best: omit it and take the SDK default. Do not invent one like "chat"	the span reaches Tempo but the UI classifies gen_ai.operation.name as unknown → the conversation renders a synthetic generation node with no attached span → the trace does not show in the conversation and the "T" (trace) icon is absent, even though trace_id/span_id are set. Silent, like #1	"Agent Observability architecture and ingest model", "Implementation rules"
7	parent_generation_ids set on multi-agent / fan-in generations	no dependency DAG; upstream eval failures don't propagate	"Multi-agent dependency tracking"
8	Workflow steps emitted for agentic pipelines with non-LLM nodes	execution graph invisible; node input/output state lost. Use the adapter if one exists, else enqueue_workflow_step; never both for one node (duplicates)	"Workflow step instrumentation (agentic pipelines)"
9	Env vars are AGENTO11Y_* (not legacy SIGIL_*); client built config-free when env present	drift; duplicated config	"Environment"
10	Content-capture mode intentional (SDK default no_tool_content); no secrets in tags/metadata/user_id	PII/secrets leak into exports	"Content capture", "Tags, metadata, and user id"
11	Client tags low-cardinality; end-user identity via user_id, not a tag	high-cardinality tags blow up metric labels	"Tags, metadata, and user id", "Implementation rules"

Step 4 — Recommend (the report)

Emit the report using llms.txt's output contract (its "Output contract" section): top opportunities first, and per opportunity — exact file:line, why it matters, a concrete diff proposal, a test plan, and any risk. Rank by severity: .env not taking effect (#0 — nothing lands at all) first, then missing OTel provider (metrics data loss), then broken export, then missing agent_version, then coverage gaps. Every recommendation cites a file:line. Then stop and ask before applying anything.

Step 5 — Apply, then verify (the loop)

Only after the developer confirms a diff. Bounded to ~3–4 iterations.

1. Apply the confirmed diff (Edit/Write). Order of preference: provider wrapper → framework adapter → hand-instrument the core SDK — only what exists for the language. Add/update a focused test for the changed instrumentation. Preserve flush/shutdown lifecycle. Never touch app logic. Pull exact usage from llms.txt / the per-language README / examples/getting-started/*.

2. Run the app for one turn to generate traffic (ask the developer to run it, or run it if there's a clean entrypoint and they approve). If the run can't happen — missing provider API key, no clean entrypoint, needs a full runtime — do not work around it by changing the model or adding a provider. Stop the loop, report the wiring as verified-by-construction (imports resolve, providers build, client + handler construct), and tell the developer the one thing left is to run one turn themselves with their key. A verified-by-construction result is a fine outcome.

3. Verify — two independent channels, don't conflate them. Instrumentation sends data on two separate paths, and confirming one says nothing about the other.

   First, confirm gcx reads the same tenant the app writes to. A verification against the wrong tenant is worse than no verification — an empty agents list gets misread as "data isn't landing" when it is, just elsewhere. Before drawing any conclusion from a gcx query: read the app's AGENTO11Y_ENDPOINT + AGENTO11Y_AUTH_TENANT_ID from its .env, then check gcx config current-context and that the active context points at that same stack/tenant. If it doesn't (e.g. context is local but the app writes to a Cloud stack), switch context or ask the developer to gcx login to the right stack — do not guess the login command. (If the gcx token is merely expired, that blocks Step 5 verification only, not writing the code — say so and continue.)

   Channel A — generations (the SDK ingest client → /api/v1/generations:export). Carries the prompt, response, tokens, cost, model, finish_reason. This is what gcx can read.

   • Via gcx: gcx agento11y agents list (does the agent appear?); gcx agento11y agents get <agent-name> (is generation_count climbing?). To find the run's conversation, either gcx agento11y conversations list --limit <n> (most-recent first, no filters — the quickest post-run check) or gcx agento11y conversations search --filters 'agent = "<agent-name>"' (--filters alone is enough; --from/--to, RFC3339, are optional and only needed to narrow a busy tenant). Then gcx agento11y conversations get <conversation-id> or gcx agento11y generations get <generation-id> — check tokens, finish reason, cost, and (for a multi-agent pipeline) that parent_generation_ids reproduce the DAG. This proves generation ingest + set_result are wired. It does NOT prove OTel is wired.

   Channel B — OTel spans/metrics (the TracerProvider/MeterProvider → OTLP exporter → /v1/traces, /v1/metrics). Carries latency/token/cost metrics. This is checklist #1, the #1 silent failure — and gcx cannot see it (traces/metrics land in the stack's Tempo/Prometheus, not the Agent Observability ingest API). So verify Channel B separately, by the strongest signal available, in this order:

   • By construction (always do this): confirm in the applied code that both a TracerProvider and a MeterProvider are created before the SDK client and shut down after it. This is static but reliable — a missing/late/no-op MeterProvider is exactly checklist #1.
   • At runtime, if you can observe it: run the real app (not an isolated probe script) with the OTel/urllib3 debug log on, and confirm you see both POST …/v1/traces → 2xx and POST …/v1/metrics → 2xx. The SDK emits spans automatically from start_generation/end (one traces POST per generation) and metrics on an interval — a clean shutdown flush surfaces both. Traces and metrics are separate exports: seeing only /v1/metrics does NOT mean traces work, and vice-versa. Never claim "traces/metrics verified" from a probe that only exercised one of them — that is the exact trap that reports Channel B as done when half of it was never sent. If you write a throwaway verification script, it must build the TracerProvider and MeterProvider and record a real generation, or just instrument the app and read its debug output.
   • In the UI: the stack's Performance / metrics view populates from metrics; traces land in the stack's Tempo (Explore → Tempo, filter by service.name). If conversations appear (Channel A) but Performance is empty, the MeterProvider is missing or no-op → back to checklist #1. Do not report OTel as wired on the strength of generations get alone, nor on metrics alone.
   • Trace shows in Tempo but NOT inside the conversation (no "T" icon): the span is landing but gen_ai.operation.name is an unrecognized value (e.g. "chat") → the UI classifies it as unknown and can't attach it to the conversation node. This is checklist #6b — fix operation_name to a recognized value (or omit it for the default) and re-run.

4. If a signal is missing, diagnose the next gap from what the checks showed, propose the fix, and loop back to step 1. After ~3–4 iterations without full signal, stop and report exactly what lands, what doesn't, and what to check next (app stderr for agento11y: warnings, credentials).

Step 6 — Hand off

Once generations land and metrics populate, instrumentation is done — that's the prerequisite for everything else. Point the developer at the next step: agento11y-test-starter to build an offline test suite for the agent (useful before shipping and for regression-testing new versions once it's live), and agento11y-prod-setup to set up online eval rules + guards on real traffic once it's deployed. The split is offline test suite vs online rules/guards — not before-traffic vs after-traffic.

Note — keeping this skill in sync

The SDK reference (env vars, provider snippets, field lists, workflow-step schema, adapter matrix) is intentionally not duplicated here — it lives in agento11y's llms.txt "Path B" and the per-language READMEs, which are the shipped source of truth. This skill holds only decision logic (state classification + gap checklist + the gcx verification loop). When a user-facing semantic changes (new SDK field, renamed env var, new framework adapter), update llms.txt (and its onboarding wizard copy); this skill points at llms.txt by section heading, not line number (line numbers drift as llms.txt is edited), so no re-pointing is needed unless a heading itself is renamed. If you find yourself pasting a provider snippet into this file, stop — fetch llms.txt instead. The references/instrumentation.md fallback is deliberately minimal for the same reason.