SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsmaple-agent-tracing-smolagents

maple-agent-tracing-smolagents

Agents & models

View repository

Updated 10-16 00:05

About this skill

Maple agent tracing: smolagents

Goal: every conversation the app runs through a smolagents agent shows up in Maple Agent Sessions as ONE session, with the transcript, each model call (model, tokens), each tool call (name, arguments, result, failure) and one lane per managed agent.

All spans come from openinference-instrumentation-smolagents.

Step 0: Detect versions and existing OpenTelemetry

  • Read pyproject.toml / requirements*.txt / uv.lock / poetry.lock. Need smolagents>=1.26 and Python >=3.10. If older, upgrade smolagents first.
  • Find which model class the app uses (OpenAIServerModel/OpenAIModel, LiteLLMModel, InferenceClientModel, AzureOpenAIModel, AmazonBedrockModel, TransformersModel...). Note any custom Model subclass that overrides generate: it will produce NO model spans.
  • Find every agent.run(...) call site and how conversations are identified (chat id, thread id, session row).
  • Search for an existing TracerProvider, trace.set_tracer_provider, opentelemetry-instrument, logfire.configure, phoenix.otel.register, langfuse, or SmolagentsInstrumentor already present. If a provider exists, REUSE it: add Maple's exporter and the processor below to it, pass it to instrument(). Never create a second provider. If SmolagentsInstrumentor().instrument() already runs, change that call instead of adding another (a second call is a silent no-op).
  • Search for openinference-instrumentation-openai, openinference-instrumentation-litellm, OpenAIInstrumentor, LiteLLMInstrumentor. If present only to trace smolagents' model calls, remove them (they double every model span).

Step 1: Ingest key and region

  • US: https://ingest.maple.dev. EU: https://ingest.eu.maple.dev.
  • Header: Authorization=Bearer <key>. Protocol: http/protobuf.
  • Key given in the prompt: use it. No key: use the literal MAPLE_TEST (ingest accepts and discards it) and tell the user to replace it with their key from Settings → Ingestion.
  • Never put a private maple_sk_ key in browser code.
  • Follow the repo's existing secret/env convention (.env, settings module, secret manager) if it has one. Otherwise inlining the key is acceptable: ingest keys are write-only.
  • App loads .env (load_dotenv()): call it at the top of tracing.py, before the provider is built. Otherwise the exporter silently targets localhost:4318 with no key.
  • Key missing: instrumentation must never crash or block the app. In tracing.py, after any load_dotenv(), when OTEL_EXPORTER_OTLP_HEADERS is unset, log one warning (logging.getLogger(__name__).warning("OTEL_EXPORTER_OTLP_HEADERS (Maple ingest key) is not set; Maple telemetry export is disabled")) and skip the provider and exporter setup. Never raise or exit over the key, and never send a header without one (opaque 401).

Step 2: Install and initialize

pip install "smolagents[openai]>=1.26" "openinference-instrumentation-smolagents>=0.1.40" \
  "opentelemetry-sdk>=1.45" "opentelemetry-exporter-otlp-proto-http>=1.45"

Use the repo's package manager (uv add, poetry add...). Use [litellm] instead of [openai] for LiteLLMModel. Do NOT use the smolagents[telemetry] extra: it installs the Arize Phoenix server.

Environment (in the repo's env mechanism):

OTEL_SERVICE_NAME=<service name, e.g. the app/package name>
OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=<env>
OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <key>"

Base URL only. OTLPSpanExporter() with no args appends /v1/traces. If you pass endpoint= in code, it must end in /v1/traces.

Create tracing.py (adapt the module path to the repo layout):

# tracing.py
import json

from openinference.instrumentation import TraceConfig
from openinference.instrumentation.smolagents import SmolagentsInstrumentor
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import SpanProcessor, TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor


class SmolagentsForMaple(SpanProcessor):
    def on_start(self, span, parent_context=None):
        if span.instrumentation_scope.name != "openinference.instrumentation.smolagents":
            return
        attrs = span.attributes
        if span.name.endswith(".run"):
            # "weather_worker.run" -> gen_ai.agent.name "weather_worker", so sub-agents get lanes
            span.set_attribute("gen_ai.agent.name", span.name.removesuffix(".run"))
        elif "tool.name" in attrs:
            # Every @tool span is named "SimpleTool"; name it after the tool instead
            span.update_name(f"execute_tool {attrs['tool.name']}")
            # The GenAI dual-write copies the tool's input schema here; record the call's arguments
            if attrs.get("input.value", "").startswith("{"):
                call = json.loads(attrs["input.value"])
                span.set_attribute("gen_ai.tool.call.arguments", json.dumps(call["kwargs"] or call["args"]))


provider = TracerProvider()  # reads OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES
provider.add_span_processor(SmolagentsForMaple())
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)

SmolagentsInstrumentor().instrument(
    tracer_provider=provider,
    config=TraceConfig(enable_genai_semconv=True),
)
  • import tracing at the top of every entry point (web app module, worker, CLI main) so instrument() runs before the first agent.run(). Import order relative to smolagents does not matter.
  • Existing provider: skip the TracerProvider()/set_tracer_provider lines, add SmolagentsForMaple() and the exporter to the existing provider, pass it as tracer_provider=.
  • enable_genai_semconv=True: required (emits the standard GenAI attributes the processor relies on). The env var OPENINFERENCE_ENABLE_GENAI_SEMCONV=true is equivalent only if set before TraceConfig is constructed; prefer the code form.
  • Keep SmolagentsForMaple exactly: it must run in on_start (before the dual-write, which never overwrites existing keys).

Step 3: One session per conversation

smolagents has no conversation id; each agent.run() is its own trace. Maple reads session.id, and the instrumentor sets it only inside using_session:

from openinference.instrumentation import using_session
from smolagents import OpenAIServerModel, ToolCallingAgent

# One agent per conversation. In a long-running server, evict idle ones or rebuild them from stored history.
agents: dict[str, ToolCallingAgent] = {}


def handle_message(conversation_id: str, text: str) -> str:
    agent = agents.get(conversation_id)
    if agent is None:
        agent = agents[conversation_id] = ToolCallingAgent(
            tools=[get_weather, calculate],
            model=OpenAIServerModel(model_id="gpt-4o-mini"),
            name="assistant",
        )
    with using_session(conversation_id):
        return str(agent.run(text, reset=False))
  • Wrap EVERY agent.run() call site in with using_session(<conversation id>):. Use the app's stored conversation/chat/thread id. Never a fresh UUID per request, never a constant.
  • If the app already tracks a user, using_attributes(session_id=..., user_id=...) also works.
  • With agent.run(..., stream=True), iterate the generator INSIDE the with block.
  • One agent object (or one persisted memory) per conversation. A module-level agent with reset=False merges every user's memory and traces into one conversation; fix it if you find it, and tell the user.
  • The id is a contextvar, not baggage. Your own spans (plain OTel tracer) don't get it; give them attributes=dict(get_attributes_from_context()) if you add any.

Step 4: Content

  • On by default: model spans carry the full input message list (system prompt, task, history, tool results) and the reply; tool spans carry arguments and results. Leave it on unless the user or repo says prompts are sensitive.
  • To turn off: TraceConfig(enable_genai_semconv=True, hide_inputs=True, hide_outputs=True) (or OPENINFERENCE_HIDE_INPUTS=true / OPENINFERENCE_HIDE_OUTPUTS=true). Narrower: hide_input_text, hide_output_text, hide_llm_invocation_parameters.
  • The hide switches do NOT mask the run span's smolagents.task attribute (holds the previous run's task text). If PII matters, drop it in a Collector (attributes processor) and tell the user.
  • Payload: default system prompt is ~3.1k chars (ToolCallingAgent) / ~8.5k (CodeAgent) and with reset=False each model span repeats the whole history. Ingest limit is 20 MiB per request; nothing to configure.

Step 5: Tools, errors, managed agents

  • Give every agent (top-level and managed) a distinct name=. Unnamed agents become ToolCallingAgent/CodeAgent and share one lane.
  • Tool exceptions are marked failed automatically (tool span status ERROR with the exception message). Do not catch exceptions inside tools just to return an error string: that hides the failure (span stays OK).
  • smolagents prompts the model to retry after every tool error until max_steps, then asks for an answer without tools (hallucination risk). If a tool can fail permanently, suggest a step_callbacks hook that tells the model to stop after the first failure; don't add it unasked.
  • Managed agents appear as <name>.run spans under the manager's Step N span; the processor names their lanes. Parallel tool/agent calls (max_tool_threads) keep context; nothing to do.
  • CodeAgent with executor_type other than local (e2b, docker, modal, ...) runs tools outside the process: no tool spans. Tell the user; nothing to fix in tracing.
  • Known, not fixable here: no gen_ai.tool.call.id on tool spans; a failed tool also marks its Step N span ERROR (Maple: toolErrorCount 1, plus an "Other errors" warning for the Step span).

Known behavior (tell the user when relevant)

  • ToolCallingAgent: a model reply that calls final_answer together with another tool raises AgentExecutionError; that step shows as a failed Step span although the run continues. It's the framework rejecting the model output, not the user's code.
  • A tool failure puts two exception events on the enclosing Step N span (wrapped AgentToolExecutionError).
  • CodeAgent with LocalPythonExecutor: tool calls from generated code produce normal tool spans; positional args are recorded as a JSON array (["Berlin"]).
  • Tokens: only input/output tokens (gen_ai.usage.input_tokens/output_tokens + llm.token_count.*); cached and reasoning tokens are not recorded. Model = the requested id (gen_ai.request.model), not the provider's response model.
  • Provider comes from the model class: OpenAIServerModel is always openai, even for an Anthropic model behind OpenRouter. OpenAIServerModel is an alias of OpenAIModel, so spans are OpenAIModel.generate.
  • Streaming (stream_outputs=True) → <ModelClass>.generate_stream spans; smolagents requests stream_options={"include_usage": True} for OpenAIServerModel, LiteLLMModel, InferenceClientModel, so tokens are kept.
  • Cost: smolagents records none; sessions show as unpriced.

Step 6: Flush

  • TracerProvider flushes on normal interpreter exit (atexit). That covers servers and CLIs that exit normally.
  • Serverless handlers (Lambda, Cloud Run jobs, Modal functions...): call provider.force_flush() in a finally before returning.
  • Scripts that may be killed or call os._exit, and notebooks: provider.force_flush() after each run; provider.shutdown() at the very end.

Step 7: Verify

Run one real conversation (2-3 messages, same conversation id, at least one tool call), and one message in a second conversation. No scriptable entry point (server, UI, REPL only): write a small driver for this run (one conversation id, 2+ turns, one tool call, flush before exit).

Without Maple access (MAPLE_TEST, no MCP): the run must exit with no export errors on stderr (Failed to export, 401 lines) AND a local exporter (provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter())) in a scratch run) must show <agent name>.run, model and execute_tool <name> spans carrying session.id. Silence alone proves nothing (no spans is silent too). Say so and list what the user should check in Maple. With the Maple MCP: list_agent_sessions with search=<conversation id> returns one row.

Otherwise check in Maple Agent Sessions (https://app.maple.dev/agent-sessions, EU app.eu.maple.dev), filtered to the service name:

  • Exactly one session per conversation id (two here), not one per message. Framework shows smolagents.
  • The first conversation has one turn per agent.run(); each turn's root span is <agent name>.run.
  • Transcript is non-empty (user text appears after New task:; tool results as tool-response messages).
  • Model calls OpenAIModel.generate (or <ModelClass>.generate[_stream]) have a model and non-zero input/output tokens, including streamed calls.
  • Session token totals equal the sum of the model calls (not 2x, not growing faster each turn).
  • Tool calls are execute_tool <tool name> with JSON arguments (the actual call's, not a schema) and results. execute_tool final_answer at the end of each turn is expected, and Maple's tool-call count includes one per agent run (managed agents too).
  • A tool that raised is counted as failed; tools that returned are not.
  • Managed agents (if any) each have their own lane named after the agent.
  • Cost shows as unpriced (smolagents records no cost). Expected.
  • Each model call appears once (no nested duplicate model span).

If sessions are split per message: using_session missing or id changing. Nothing arrives: exporter endpoint/header wrong, .env loaded after tracing.py, or process exited without flushing. 401 ingest_unauthorized / "Invalid ingest key" with a key you trust: keys are region-bound, so it likely belongs to the other region; try the other endpoint.

Do not

  • Do not use phoenix.otel.register() (the smolagents docs' setup) to send to Maple: it skips SmolagentsForMaple (no lanes, tools named SimpleTool).