maple-agent-tracing-opentelemetry

Maple agent tracing: OpenTelemetry GenAI conventions (any language)

Goal: every conversation = one Maple Agent Session. Each user message = one trace rooted at an invoke_agent span, with a chat span per model call (model, tokens, transcript) and an execute_tool span per tool call (args, result, failures).

Mechanism: you write the spans. Maple classifies a span only by gen_ai.operation.name, groups a trace by gen_ai.conversation.id, and reads content only from span attributes. Hand-written spans show as framework "Unidentified" (vendor unknown:genai); that is expected.

Step 0: Detect

Language and entry points (web server, workers, scripts, serverless handlers).
Is a supported framework the real agent runtime? (@mastra/core, ai, agents/@cloudflare/ai-chat, genkit/@genkit-ai/*, @openai/agents/openai-agents, langchain/langgraph, pydantic-ai, crewai, google-adk, llama-index, strands-agents, smolagents, agno, dspy, haystack-ai, agent-framework, Spring AI, litellm, Claude Agent SDK). If yes, stop and use maple-agent-tracing-<framework> instead; use this skill only for the parts that framework doesn't cover, or for the maple_ai.session.id wrapper (Step 4).
Existing OTel setup. Search for TracerProvider, NodeTracerProvider, NodeSDK, registerOTel, set_tracer_provider, opentelemetry-instrument, logfire.configure, sentry_sdk.init/Sentry.init, otel.SetTracerProvider. Exists → add Maple's exporter/processor to it; never create a second provider.
Existing GenAI auto-instrumentation on the model client (@opentelemetry/instrumentation-openai, opentelemetry-instrumentation-openai-v2, OpenLLMetry Traceloop.init, OpenInference OpenAIInstrumentor, logfire.instrument_openai). Pick one source of chat spans: either keep that instrumentation (then see maple-agent-tracing-provider-sdks) or remove it and write chat spans here. Both = every model call twice.
Find in the code: the agent loop (where one user message is handled), every model call site, every tool dispatch, sub-agent calls, and where the conversation/chat/thread id lives in the request.

Step 1: Key and region

US: https://ingest.maple.dev. EU: https://ingest.eu.maple.dev.
Header: Authorization=Bearer <key>.
Key given in the prompt → use it.
No key → use the literal MAPLE_TEST (ingest accepts and discards it) and tell the user to replace it with their key from Settings → Ingestion.
Never put a private maple_sk_ key in browser code.
Follow the repo's secret/env convention if it has one. Otherwise inline is acceptable: ingest keys are write-only.

OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <key>"
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

The exporters append /v1/traces.

The SDK exporters read these env vars when the exporter is constructed. If the app loads .env (dotenv, load_dotenv(), --env-file), load it at the top of the tracing module, before the provider is built; otherwise the exporter silently targets localhost:4318 with no key.
If the header is built from your own env var and it is unset, log one warning (MAPLE_INGEST_KEY is not set; Maple telemetry export is disabled) and skip the Maple exporter so the app runs normally. Never throw or exit over the key, and never let it become Bearer undefined (opaque 401) or a bare KeyError on import. Or inline the key when the repo has no env convention.
401 ingest_unauthorized / "Invalid ingest key" with a key you trust: keys are region-bound, so it usually belongs to the other region. Try the other endpoint.

Step 2: Install + init

Read the reference for the language and adapt it:

TypeScript/Node: references/typescript.md
Python: references/python.md
Other languages: the language's OTel SDK with an OTLP/HTTP exporter, following the steps below.

Rules:

Init module is imported first in every entry point. Set a real service.name and deployment.environment.name.
Name the tracer after the app (e.g. support-agent). Never openrouter, langsmith, litellm, haystack, ai, gen_ai: Maple fingerprints frameworks by scope name and would treat your spans as that framework's.
Keep the project's loop structure; add spans around its existing calls. Use the reference's complete loop only when there is no loop yet.

Step 3: The three spans (exact keys)

invoke_agent (kind INTERNAL, name invoke_agent <agent>), around one agent run; for a user turn it is the trace root:

gen_ai.operation.name=invoke_agent, gen_ai.agent.name, gen_ai.conversation.id (Step 4)
optional: gen_ai.input.messages (the user message), gen_ai.output.messages (final answer)

chat (kind CLIENT, name chat <model>), around each model call:

at start: gen_ai.operation.name=chat (or generate_content/text_completion), gen_ai.provider.name, gen_ai.request.model, gen_ai.system_instructions, gen_ai.input.messages
at end: gen_ai.response.id, gen_ai.response.model, gen_ai.response.finish_reasons (string array), gen_ai.output.messages, usage (Step 7), gen_ai.response.time_to_first_chunk (double, SECONDS, streamed calls)

execute_tool (kind INTERNAL, name execute_tool <tool>), around each tool call:

gen_ai.operation.name=execute_tool, gen_ai.tool.name (real name), gen_ai.tool.call.id (the model's call id), gen_ai.tool.type=function
gen_ai.tool.call.arguments: JSON string of an object
gen_ai.tool.call.result: JSON string of the tool's return value; a string return can be set as its plain text.

Message JSON (input.messages/output.messages): array of {role, parts}; parts {type:"text",content}, {type:"tool_call",id,name,arguments:<object>}, {type:"tool_call_response",id,response}, {type:"reasoning",content}. Output messages add finish_reason. system_instructions = array of parts, no role: [{"type":"text","content":"..."}]. Always a JSON string attribute; plain-text messages don't render.

Also read: a message may carry content (string or part array) instead of parts. gen_ai.response.model wins over gen_ai.request.model when both are set.

Legacy spellings are read as fallbacks (use current names in new code): gen_ai.system (→ gen_ai.provider.name, renamed in semconv 1.37), gen_ai.usage.prompt_tokens/completion_tokens, whole-value gen_ai.prompt/gen_ai.completion, gen_ai.usage.cache_creation.input_tokens (→ cache_write), gen_ai.usage.total_cost (→ cost).

provider.name = the API actually called: openai, anthropic, gcp.gemini, gcp.vertex_ai, aws.bedrock, azure.ai.openai, mistral_ai, groq, x_ai, deepseek, or openrouter for OpenRouter.

Step 4: Session id (required)

The id is read only from spans that have gen_ai.operation.name; on an unclassified span it is ignored.
Set gen_ai.conversation.id on the turn's invoke_agent span from the app's conversation/chat/thread id. Same value for every message of a conversation; different across conversations. One classified span per trace is enough; every span in the trace joins.
Never: uuid4()/randomUUID() per request, the trace id, a module-level constant, a per-process default. No real id (single-shot script) → generate one per conversation, not per message, and reuse it.
Sub-agents in the same trace: no id (they inherit the trace's session) or the same id. Two different ids in one trace → the lexically larger silently wins.
Chat backends: one message list per conversation (keyed by that id, persisted), never one global list.

Escape hatch, only when a framework's spans carry a session key Maple ignores for that framework (LiteLLM, Haystack): wrap each turn in your own span with gen_ai.operation.name=invoke_agent, gen_ai.agent.name, and maple_ai.session.id=, and run the framework inside it. Rules:

Only on your own wrapper span, never on framework spans (it re-vendors the span to maple: framework decoding lost, usage read as inclusive of cache).
Use the same value the framework would use for the session.
Not needed for hand-written spans: use gen_ai.conversation.id.
The session then shows framework Maple.

// chatId comes from your request; frameworkAgent is the framework's agent
await tracer.startActiveSpan(
	"invoke_agent support",
	{
		attributes: {
			"gen_ai.operation.name": "invoke_agent",
			"gen_ai.agent.name": "support",
			"maple_ai.session.id": chatId,
		},
	},
	async (span) => {
		try {
			return await frameworkAgent.run(message) // the framework's spans nest under this one
		} finally {
			span.end()
		}
	},
)

Step 5: Content

Content = gen_ai.system_instructions, gen_ai.input.messages, gen_ai.output.messages (chat), gen_ai.tool.call.arguments, gen_ai.tool.call.result (execute_tool). On span attributes only: span events, log records, and indexed keys (gen_ai.prompt.0.content, llm.input_messages.0.*) are not read.
Do not set OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT / OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT; if the platform sets one, unset it. Truncated JSON is dropped whole. To cap size, drop the oldest messages whole before serializing.
Replace base64 images/files in history with a placeholder part (ingest rejects requests > 20 MiB with 413).
User wants no content → skip those five attributes (everything else still works; transcript empty). Wants redaction → redact inside toSemconv/to_semconv before serializing, or an OTel Collector redaction/transform processor.
Never put API keys or Authorization headers in any attribute.

Step 6: Tools, errors, sub-agents

Tool failure: set status ERROR with the error message as description, set error.type (exception class or error code), no gen_ai.tool.call.result, then return the error to the model as the tool result so the loop continues. Keep the message specific: Maple's tool pages group failures by it.
Maple counts any span as failed if it has status ERROR, a non-empty error.type, or gen_ai.response.status=failed.
Tools that return {"error": ...} instead of raising: mark the span failed the same way when you detect it.
Model call failure: status ERROR + error.type (HTTP status or exception class), rethrow.
Agent run failure: same on invoke_agent.
Sub-agent: call its loop inside the delegating tool's execute_tool span, so execute_tool ask_x → invoke_agent x → its chat/execute_tool. Distinct gen_ai.agent.name per agent (lanes need it).
Delegation detection: an execute_tool span whose only child is an invoke_agent span is drawn as a delegation into a lane named after the child's gen_ai.agent.name; the tool's arguments/result become the lane's input/output. Two agents with the same name share one lane; an invoke_agent span without a name gets no lane.
Parallel tools: start each execute_tool span inside the turn's context (Node Promise.all keeps it; Python threads need contextvars.copy_context().run).

Step 7: Tokens and cost

Usage on chat spans only, never cumulative totals on invoke_agent.
Keys: gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cache_read.input_tokens, gen_ai.usage.cache_write.input_tokens, gen_ai.usage.reasoning.output_tokens (ints). total_tokens is not read.
Totals as the spec defines them: input_tokens = every prompt token, cache reads and writes included; output_tokens = every completion token, reasoning included. OpenAI, OpenRouter and Gemini's promptTokenCount already count that way. Anthropic's input_tokens excludes cache: send input_tokens + cache_read_input_tokens + cache_creation_input_tokens. Gemini's candidatesTokenCount excludes thoughts: send candidatesTokenCount + thoughtsTokenCount.
Streaming OpenAI-compatible: stream_options: {include_usage: true}. OpenAI sends usage in an extra last chunk with empty choices; OpenRouter always sends usage + cost on the chunk carrying finish_reason. Read chunk.usage before skipping chunks without choices.
OpenRouter (Claude included): prompt_tokens already includes cached_tokens → provider openrouter, copy as is. Optional: prompt_tokens_details.cache_write_tokens → gen_ai.usage.cache_write.input_tokens.
Cost: gen_ai.usage.cost (double, USD) on chat spans. OpenRouter returns usage.cost → copy it. Other providers return none → compute only if the project has a price table; otherwise leave it (Maple shows "unpriced"; it never prices tokens).
gen_ai.response.id always (dedupes against gateway mirrors such as OpenRouter Broadcast).

Step 8: Flush

Node script/CLI: await provider.shutdown() in finally. Serverless: await provider.forceFlush() before returning (inside waitUntil/after() if available). Both reject when an export failed: add .catch((err) => console.error("telemetry flush failed", err)) so a Maple outage can't crash the app. Long-running server: flush on SIGTERM, nothing per request.
Python script: provider.shutdown() in finally. Lambda: force_flush() in finally. Notebooks/workers: force_flush() per cell/task.

Step 9: Verify

If the app has no scriptable entry point (server, REPL, UI only), write a small driver for this run: one conversation id, 2+ turns, at least one tool call, flush before exit.

Run one real conversation: 2+ messages with the same id, one streamed reply if the app streams, one tool call, one failing tool if one exists, one sub-agent call if the app delegates; then a second conversation. Wait ~30 s; Maple → Agent Sessions, filter by service name. Check:

One session per conversation; session id = the conversation id; second conversation = different session; no trace:<id> sessions.
One turn per user message, labeled with it; transcript shows system instructions, user messages, assistant replies, tool calls and results.
Spans invoke_agent <agent> → chat <model> / execute_tool <tool>, all in the turn's trace (no orphan roots).
Every chat span: model, provider, response id, input+output tokens (streamed ones too), finish reasons; TTFT in seconds on streamed calls.
Tool spans: real names, call ids matching the model's tool calls, JSON args and results.
The failing tool is failed with its message; successful tools and model calls are not.
Sub-agents: own lane with their gen_ai.agent.name, same session.
Cost present only if spans carry gen_ai.usage.cost; else "unpriced".
No attribute contains an API key, Bearer , sk-or-, maple_sk_.

With the Maple MCP: list_agent_sessions with search=<conversation id> returns one row.

Check without Maple access: the run exits with no export errors on stderr (Failed to export, OTLPExporterError, 401 lines) AND a temporary console exporter (ConsoleSpanExporter + SimpleSpanProcessor) shows the expected span tree with gen_ai.conversation.id, and every messages attribute parses with JSON.parse/json.loads. Silence alone proves nothing: no spans also looks silent.