SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsmaple-agent-tracing-llamaindex

maple-agent-tracing-llamaindex

Agents & models

View repository

Updated 10-16 06:45

About this skill

Maple agent tracing: LlamaIndex (Python)

Goal: every conversation = one Maple Agent Session. Each agent.run() / workflow.run() = one turn (one trace) with transcript, one model span per model call with tokens, FunctionTool.acall tool spans with results, failed tools marked failed, sub-agents in their own lanes.

Mechanism: openinference-instrumentation-llama-index (scope openinference.instrumentation.llama_index). Maple reads the session (session.id from using_session), transcript, tokens and tool calls from its spans. TraceConfig(enable_genai_semconv=True) is recommended: it also emits standard GenAI attributes (gen_ai.*, incl. gen_ai.conversation.id).

Step 0: Detect

  1. Versions: python -c "import llama_index.core as c; print(c.__version__)" (or pyproject.toml / uv.lock / requirements*.txt).
    • Need llama-index-core >= 0.14.19 (tested 0.14.25). Older: the instrumentor logs DependencyConflict and does nothing. Tell the user to upgrade.
    • Python >= 3.10.
  2. Existing tracing. Search for LlamaIndexOpenTelemetry, llama_index.observability.otel, LlamaIndexInstrumentor, set_global_handler, TracerProvider(, set_tracer_provider, opentelemetry-instrument, logfire.configure, sentry_sdk.init, langfuse, phoenix.otel.register.
    • LlamaIndexOpenTelemetry (native llama-index-observability-otel) present → replace it with Step 2 (it puts content in span events, loses reply and usage on streamed calls, ignores OTEL_EXPORTER_OTLP_*). Never run both: every span doubles. Ask before removing if it also feeds another backend.
    • LlamaIndexInstrumentor already used (Phoenix/Langfuse/Arize) → keep it, reuse its provider, add config=TraceConfig(enable_genai_semconv=True) and the Maple processor to that provider.
    • Another TracerProvider exists → add the Maple processor chain to it; do NOT create a second provider.
  3. Other instrumentors on the model client (openinference-instrumentation-openai, -litellm, OpenLLMetry Traceloop.init, logfire.instrument_openai) → duplicate model spans with their own usage. Remove for LlamaIndex models (ask if they serve other code).
  4. Find: every agent.run( / workflow.run( / AgentWorkflow( call, where the chat/thread id lives in the request, every FunctionAgent(/ReActAgent(/CodeActAgent( construction, every tool that runs another agent, every ctx.wait_for_event( (HITL).
  5. LLM class: OpenAILike/OpenRouter need is_function_calling_model=True or tools silently never run (no tool spans). Check it's set.

Step 1: Key and region

  • US: https://ingest.maple.dev. EU: https://ingest.eu.maple.dev.
  • Header: Authorization=Bearer <key>.
  • Key given in the prompt → use it.
  • No key → use the literal MAPLE_TEST (ingest accepts and discards it) and tell the user to replace it with their key from Settings → Ingestion.
  • Never put a private maple_sk_ key in browser code.
  • Follow the repo's secret/env convention (.env, settings module, secret manager) if it has one. Otherwise inline is acceptable: ingest keys are write-only.
  • Building the header in code from an env var: when it is unset, log one warning (MAPLE_INGEST_KEY is not set; Maple telemetry export is disabled) and skip the Maple exporter so the app runs normally. Never raise or exit over the key, and never send Bearer None (opaque 401) or hit a bare KeyError on import. Or inline the key when the repo has no env convention.
  • 401 ingest_unauthorized / "Invalid ingest key" with a key you trust: keys are region-bound, so it usually belongs to the other region. Try the other endpoint.

Env vars (OTLPSpanExporter() reads them and appends /v1/traces):

OTEL_SERVICE_NAME=support-agent
OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production
OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <key>"
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

If you pass OTLPSpanExporter(endpoint=...) in code, it must end in /v1/traces (no auto-append).

Step 2: Install + init

Add with the repo's package manager (uv/poetry/pip):

pip install "llama-index-core>=0.14.25" "openinference-instrumentation-llama-index>=4.5.2" \
  "opentelemetry-sdk>=1.45" "opentelemetry-exporter-otlp-proto-http>=1.45"

Create tracing.py verbatim (adapt nothing except where noted):

# tracing.py
from llama_index.core.instrumentation.dispatcher import active_instrument_tags
from openinference.instrumentation import TraceConfig
from openinference.instrumentation.llama_index import LlamaIndexInstrumentor
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import SpanProcessor, TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

LLM_METHODS = (".chat", ".achat", ".stream_chat", ".astream_chat",
               ".complete", ".acomplete", ".stream_complete", ".astream_complete")


class LlamaIndexForMaple(SpanProcessor):
    """Sits in front of the exporter: one span per model and tool call, agent names, no false HITL failures."""

    def __init__(self, exporter_processor: SpanProcessor):
        self._next = exporter_processor
        self._open_llm_spans = {}

    def on_start(self, span, parent_context=None):
        # instrument_tags({"gen_ai.agent.name": ...}) becomes an attribute, so sub-agents get lanes
        agent_name = active_instrument_tags.get().get("gen_ai.agent.name")
        if agent_name:
            span.set_attribute("gen_ai.agent.name", agent_name)
        if span.name.endswith((".call_tool", ".aggregate_tool_results")):
            # agent workflow steps around the tool span; by name alone Maple would count them as tool calls
            span.set_attribute("gen_ai.operation.name", "invoke_workflow")
        if span.name.endswith(LLM_METHODS):
            self._open_llm_spans[span.context.span_id] = span
        self._next.on_start(span, parent_context)

    def on_end(self, span):
        self._open_llm_spans.pop(span.context.span_id, None)
        if span.name.endswith("._prepare_chat_with_tools"):
            return  # builds the request; never calls the model
        if (span.status.description or "").startswith("WaitingForEvent"):
            return  # ctx.wait_for_event() suspends the tool and replays it later; not a failure
        outer = self._open_llm_spans.get(span.parent.span_id) if span.parent else None
        if outer is not None and outer.name == span.name:
            outer.set_attributes(span.attributes)  # the inner twin holds the messages and usage
            return
        self._next.on_end(span)

    def shutdown(self):
        self._next.shutdown()

    def force_flush(self, timeout_millis=30000):
        return self._next.force_flush(timeout_millis)


provider = TracerProvider()  # reads OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES
provider.add_span_processor(LlamaIndexForMaple(BatchSpanProcessor(OTLPSpanExporter())))
trace.set_tracer_provider(provider)

LlamaIndexInstrumentor().instrument(
    tracer_provider=provider,
    config=TraceConfig(enable_genai_semconv=True),
)
  • Import tracing first in the entry point (app module, main.py, worker). instrument() must run before the first agent.run().
  • The app loads .env (load_dotenv(), --env-file): call load_dotenv() at the top of tracing.py, before the provider is built. Otherwise the exporter silently targets localhost:4318 with no key.
  • Existing provider: skip TracerProvider()/set_tracer_provider; call existing.add_span_processor(LlamaIndexForMaple(BatchSpanProcessor(OTLPSpanExporter()))) and pass tracer_provider=existing.
  • The exporter MUST be added through LlamaIndexForMaple, never directly: without it every model call counts 2-3x in Maple (_prepare_chat_with_tools + nested same-name astream_chat/achat spans, all OpenInference kind LLM) and every tool call 3x (call_tool / aggregate_tool_results step spans are classified as tools by name).
  • No service.name default is acceptable: set OTEL_SERVICE_NAME (never unknown_service).

Step 3: Session id (required)

Nothing in LlamaIndex sets a conversation id; Context and llamaindex.run_id never reach Maple as one. Wrap every agent.run(...) / workflow.run(...) CALL in using_session(<app conversation id>); add instrument_tags with the agent name in the same with:

from llama_index.core.agent.workflow import AgentStream
from llama_index.core.instrumentation.dispatcher import instrument_tags
from llama_index.core.workflow import Context
from openinference.instrumentation import using_session

contexts: dict[str, Context] = {}


async def handle_message(conversation_id: str, text: str):
    if conversation_id not in contexts:
        contexts[conversation_id] = Context(agent)
    ctx = contexts[conversation_id]
    with using_session(conversation_id), instrument_tags({"gen_ai.agent.name": agent.name}):
        handler = agent.run(user_msg=text, ctx=ctx)
    async for event in handler.stream_events():
        if isinstance(event, AgentStream):
            yield event.delta
    await handler
  • Only the run() call must be inside the with (tasks start there and inherit the contextvars). Consume the stream / await handler outside it; do not yield inside the with in async generators.
  • The id must be stable per conversation and unique across conversations: the app's chat/thread id. No uuid4() per request, no constant, no module-level default. No id available → ask the user where the conversation boundary is; for a one-shot script, one uuid per conversation reused across its turns.
  • One Context per conversation (a shared Context shares memory across users).
  • HITL: keep handler.ctx.send_event(HumanResponseEvent(...)) on the same handler; the resumed step stays in the same trace and session.
  • Do not use session.id/maple_ai.session.id attributes of your own; using_session already writes session.id (and gen_ai.conversation.id with enable_genai_semconv).

Step 4: Content

  • On by default: gen_ai.input.messages (system + history + tool results), gen_ai.output.messages (incl. tool_call parts), tool results. Maple's transcript needs them.
  • User wants content off → TraceConfig(enable_genai_semconv=True, hide_inputs=True, hide_outputs=True) (or hide_input_text/hide_output_text to keep structure). Env equivalents OPENINFERENCE_HIDE_*. Tell them the transcript will be empty.
  • Pattern redaction (emails, cards): recommend an OTel Collector redaction processor.

Step 5: Tools, errors, sub-agents

  1. Give every agent a name= and wrap each agent's run() in instrument_tags({"gen_ai.agent.name": agent.name}) (the processor copies it onto every span). No tag = no agent facet, no lanes.
  2. Multi-agent via custom Workflow: run the whole workflow inside using_session(...); inside steps call sub-agents like this:
async def run_agent(agent: FunctionAgent, message: str) -> str:
    with instrument_tags({"gen_ai.agent.name": agent.name}):
        handler = agent.run(user_msg=message)
    return str(await handler)

Parallel fan-out (several ctx.send_event calls to separate steps or to a @step(num_workers=N), joined with ctx.collect_events) stays in one trace with the session and agent tags. 3. Agents-as-tools: put the same instrument_tags block inside the tool function around the sub-agent's run(). 4. AgentWorkflow handoffs: one AgentWorkflow.run span, no per-agent spans, so no lanes; every span carries the root agent's tag and the handoff is a handoff tool call. Tag the call with the root agent's name. If the user needs lanes, suggest running agents via workflow steps or tools (ask first; it changes app behavior). 5. Tool failures: a tool that raises → FunctionTool.acall status ERROR with the exception message (Maple counts it). Tools that return "Error: ..." look successful: convert to raise only where the user agrees. 6. HITL (ctx.wait_for_event): the first, suspended FunctionTool.acall ends ERROR WaitingForEvent: ...; the processor drops it. Nothing to add. 7. Known, unfixable here: no gen_ai.tool.call.id on tool spans.

Step 6: Tokens, cost, streaming

  • Tokens come from the provider response on the model span. FunctionAgent streams by default; OpenAI-API streams only include usage when asked. For OpenAI/OpenAILike models not on OpenRouter add:
llm = OpenAI(model="gpt-4o-mini", additional_kwargs={"stream_options": {"include_usage": True}})

(LlamaIndex strips it from non-streaming requests.) OpenRouter sends usage without it.

  • Streamed model spans end when the stream is handed back (~1 ms), so their duration is not model latency (Maple's session inference time totals a few ms; FunctionAgent streams by default, so this is every call). If the app never streams tokens to users, FunctionAgent(..., streaming=False) gives real model-span durations. Ask before changing it.
  • Cost: never recorded. Sessions show "unpriced". Do not add pricing code. OpenRouter users can add OpenRouter Broadcast for cost.

Step 7: Flush

  • The provider flushes at normal interpreter exit.
  • Scripts/CLIs/one-shot jobs: provider.shutdown() in a finally at the end.
  • Serverless handlers, Celery/RQ tasks, notebooks: provider.force_flush() in a finally after each run (from tracing import provider).
  • FastAPI/long-running servers: nothing extra; optionally provider.shutdown() in the lifespan shutdown.

Step 8: Verify

Run one real conversation: 2+ messages with the same id, at least one tool call, a failing tool if one exists, and a sub-agent run if the app delegates. No scriptable entry point (server, REPL, UI only) → write a small driver: one conversation id, 2+ turns, at least one tool call, provider.shutdown() before exit. Then check (Maple → Agent Sessions, filter by service name; wait ~30-60 s):

  • Exactly one session per conversation, id = the id you passed. A second conversation is a second session. Not trace:<id> sessions.
  • One turn per run(); each trace roots at FunctionAgent.run / AgentWorkflow.run / <YourWorkflow>.run.
  • Framework shows "LlamaIndex".
  • Transcript shows user messages, assistant replies and tool calls.
  • LLM call count ≈ real number of model calls (one <ModelClass>.astream_chat/.achat span per call; no _prepare_chat_with_tools spans exported).
  • Every model span has a model and input/output tokens, including streamed calls.
  • Tool call count = real tool calls (only FunctionTool.acall; call_tool/aggregate_tool_results carry gen_ai.operation.name=invoke_workflow). Tool spans carry gen_ai.tool.name (real tool name) and a result; model and tool spans sit under the turn's agent span in the same trace.
  • A failing tool is failed with its message; successful and HITL-approved tools are not.
  • Sub-agents appear as separate lanes with their names, under the caller's session.
  • Cost shows "unpriced".
  • No attribute contains an API key or Bearer token.

Edge cases:

  • Native llama-index-observability-otel (0.7.0, LlamaIndexOpenTelemetry): model settings/prompt only in LLMChatStartEvent span events (Maple ignores events); the end event with reply+usage is dropped on streamed calls; two nested <Model>.astream_chat spans per call; ignores OTEL_EXPORTER_OTLP_* and defaults to ConsoleSpanExporter without span_exporter=. If a user insists on keeping it and only wants sessions: instrument_tags({"gen_ai.conversation.id": conversation_id}) around agent.run() groups traces (dotted tag keys become span attributes verbatim).
  • Span layout the processor fixes: BaseWorkflowAgent.run_agent_step → <Model>._prepare_chat_with_tools (kind LLM, ~1 ms) + <Model>.astream_chat (OpenAILike override) → <Model>.astream_chat (inner, holds messages+usage). Classes implementing the call directly (OpenAI) produce two spans. Merge only applies to directly nested identical names, so CondensePlusContextChatEngine.chat → OpenAI.chat is untouched. Tokens were never doubled (only innermost has usage), only call counts.
  • OPENINFERENCE_ENABLE_GENAI_SEMCONV=true is equivalent to the config flag only if set before TraceConfig is built.
  • Tool spans: FunctionTool.acall (kind TOOL, execute_tool), result is LlamaIndex ToolOutput JSON with raw_input/raw_output (real args are in raw_input). On failure the call_tool step stays OK because FunctionAgent hands the error to the model.
  • Agents-as-tools: a tool span with one FunctionAgent.run child shows as a delegation; tool args/result become the lane's input/output. Maple opens a lane for every agent span whose name differs from its caller's.
  • Provider: OpenRouter and every OpenAILike report openai, even for Anthropic models. No response model name or gen_ai.response.id recorded.
  • OpenRouter Broadcast for cost: because model spans lack gen_ai.response.id, nest Broadcast spans under them (https://maple.dev/docs/agent-tracing/openrouter#nest-broadcast-under-your-own-traces) or each call counts twice.
  • With a persistent Context every model span repeats the whole chat. Maple has no per-attribute limit; ingest accepts requests up to 20 MiB.
  • Streaming query engines on llama-index-core 0.14.25: the model call of a StreamingResponse runs after the query span ended, so it lands in a separate trace. Fix pending in OpenInference PR #3841 (https://github.com/Arize-ai/openinference/pull/3841); until released, pin llama-index-core<0.14.25 if the app traces streaming query engines. Agents unaffected.

Without Maple access, both must hold: the run exits with no export errors on stderr (Failed to export, 401 lines), AND a temporary SimpleSpanProcessor(ConsoleSpanExporter()) passed into LlamaIndexForMaple shows the agent, model and FunctionTool.acall spans with gen_ai.conversation.id identical on every span of a conversation and gen_ai.agent.name set. Silence alone proves nothing (no spans is silent too). With the Maple MCP: list_agent_sessions with search=<conversation id> returns one row.