maple-agent-tracing-microsoft-agent-framework

Maple agent tracing: Microsoft Agent Framework and Semantic Kernel

Goal: one user conversation = one Maple Agent Session, with a transcript, every model call, every tool call (arguments, results, failures) and tokens.

The framework emits the spans itself. You add: an OTLP/HTTP exporter, content capture, a gen_ai.conversation.id span processor (the framework never sets one for local-history sessions), and a flush.

Step 0: Detect

• Python MAF: agent-framework, agent-framework-core in pyproject.toml / requirements*.txt / uv.lock. Check the installed version (python -c "import agent_framework; print(agent_framework.__version__)"). Target ≥ 1.19.0; upgrade if older (1.13 lacks the otlp_* arguments used below).
• .NET MAF: Microsoft.Agents.AI in *.csproj. Target ≥ 1.22.0.
• Semantic Kernel: semantic-kernel (Python, target ≥ 1.44.1) or Microsoft.SemanticKernel (.NET). Use the Semantic Kernel section.
• Existing OpenTelemetry: search for TracerProvider(, set_tracer_provider, configure_azure_monitor, logfire.configure, configure_otel_providers, AddOpenTelemetry(, Sdk.CreateTracerProviderBuilder. If a provider exists, add Maple's exporter and the conversation processor to it; do not create a second provider and do not call configure_otel_providers().
• Find every place a user message is handled (HTTP route, queue consumer, CLI loop) and what identifies the conversation there (chat id, thread id, AgentSession). You need it in Step 3.

Step 1: Key and region

• US endpoint https://ingest.maple.dev, EU https://ingest.eu.maple.dev. Header Authorization=Bearer <key>. Protocol http/protobuf.
• Key in the user's prompt: use it. No key: use the literal MAPLE_TEST (ingest accepts and discards it) and tell the user to replace it with their key from Settings → Ingestion.
• Never put a private maple_sk_ key in browser code.
• Follow the repo's existing secret/env convention (.env, settings class, user-secrets). If there is none, inlining the ingest key is acceptable: ingest keys are write-only.
• Key from an env var: when it's unset, log one warning (MAPLE_INGEST_KEY is not set; Maple telemetry export is disabled) and skip the Maple exporter so the app runs normally (Step 2 shows it). Never raise, exit or throw over the key, no bare KeyError on os.environ['MAPLE_INGEST_KEY'], and never pass a null Environment.GetEnvironmentVariable(...) into the .NET header (Bearer with no key is an opaque 401).
• App loads .env (load_dotenv()): call it before configure_otel_providers() / the provider is built, and before reading the key. Otherwise the exporter silently targets localhost:4318 with no key.

Step 2: Install and init

Python MAF

pip install "agent-framework-core>=1.19.0" "agent-framework-openai>=1.14.4" opentelemetry-exporter-otlp-proto-http

(Keep the agent-framework meta-package if the project already uses it; still add opentelemetry-exporter-otlp-proto-http, MAF installs no exporter.)

Create maple_tracing.py next to the app entry point:

from contextlib import contextmanager
from contextvars import ContextVar

from opentelemetry.sdk.trace import SpanProcessor

_conversation_id: ContextVar[str | None] = ContextVar("conversation_id", default=None)


class ConversationIdProcessor(SpanProcessor):
    """Puts gen_ai.conversation.id on every span started inside `conversation()`."""

    def on_start(self, span, parent_context=None):
        if (conversation_id := _conversation_id.get()) is not None:
            span.set_attribute("gen_ai.conversation.id", conversation_id)


@contextmanager
def conversation(conversation_id: str):
    token = _conversation_id.set(conversation_id)
    try:
        yield
    finally:
        _conversation_id.reset(token)

At startup, once, before agents are created:

import logging
import os

from agent_framework.observability import configure_otel_providers
from opentelemetry import trace

from maple_tracing import ConversationIdProcessor

key = os.environ.get("MAPLE_INGEST_KEY")
if key:
    configure_otel_providers(
        service_name="<service-name>",
        resource_attributes={"deployment.environment.name": "<env>"},
        otlp_endpoint="https://ingest.maple.dev",
        otlp_protocol="http/protobuf",
        otlp_headers={"Authorization": f"Bearer {key}"},
        enable_sensitive_data=True,
        enable_message_events=False,
    )
    trace.get_tracer_provider().add_span_processor(ConversationIdProcessor())
else:
    # A missing key disables export; it never stops the app.
    logging.getLogger(__name__).warning("MAPLE_INGEST_KEY is not set; Maple telemetry export is disabled")

Equivalent env-var config, with a bare configure_otel_providers() call:

export OTEL_SERVICE_NAME="<service-name>"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.maple.dev"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <key>"
export ENABLE_SENSITIVE_DATA="true"
export ENABLE_MESSAGE_EVENTS="false"

• otlp_protocol is mandatory: MAF defaults to gRPC (spec says http/protobuf) and fails silently or with an ImportError.
• configure_otel_providers() appends /v1/traces, /v1/metrics, /v1/logs to the endpoint. OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, if set, is used as-is and must include /v1/traces.
• Existing provider instead: add BatchSpanProcessor(OTLPSpanExporter(endpoint="https://ingest.maple.dev/v1/traces", headers={...})) and ConversationIdProcessor() to it, then from agent_framework.observability import enable_instrumentation; enable_instrumentation(enable_sensitive_data=True, enable_message_events=False).
• For OpenRouter or any Chat Completions endpoint use OpenAIChatCompletionClient(model=..., api_key=..., base_url=...), not OpenAIChatClient (Responses API; OpenRouter rejects its previous_response_id on turn 2).

.NET MAF

dotnet add package Microsoft.Agents.AI --version 1.22.0
dotnet add package Microsoft.Agents.AI.OpenAI --version 1.22.0
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol --version 1.19.1

using var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .ConfigureResource(r => r.AddService("<service-name>"))
    .AddSource("*Microsoft.Agents.AI*")
    .AddSource("*Microsoft.Extensions.AI")
    .AddProcessor(new ConversationIdProcessor())
    .AddOtlpExporter(o =>
    {
        o.Endpoint = new Uri("https://ingest.maple.dev/v1/traces");
        o.Protocol = OtlpExportProtocol.HttpProtobuf;
        o.Headers = "Authorization=Bearer <key>";
    })
    .Build();

AIAgent agent = chatClient
    .AsAIAgent(instructions: "...", name: "<agent_name>", tools: [...])
    .AsBuilder()
    .UseOpenTelemetry(configure: a => a.EnableSensitiveData = true)
    .Build();

sealed class ConversationIdProcessor : BaseProcessor<Activity>
{
    public static readonly AsyncLocal<string?> Current = new();

    public override void OnStart(Activity activity)
    {
        if (Current.Value is { } id) activity.SetTag("gen_ai.conversation.id", id);
    }
}

• Default source names carry an Experimental. prefix; AddSource("Microsoft.Agents.AI.*") matches nothing. Keep the leading *.
• UseOpenTelemetry() on the agent (1.22) also instruments its chat client. Do not add a second UseOpenTelemetry() on the IChatClient.
• Name every tool: AIFunctionFactory.Create(GetWeather, name: "get_weather"). Without name:, a local function in top-level Program.cs is exported as _Main_g_GetWeather_0_3.
• Workflows: .WithOpenTelemetry() on the WorkflowBuilder (source Microsoft.Agents.AI.Workflows, matched by the wildcard).
• In a hosted app, put the same sources, processor and exporter in builder.Services.AddOpenTelemetry().WithTracing(...).

Step 3: Conversation id

MAF only sets gen_ai.conversation.id from the provider's own conversation id (AgentSession.service_session_id: Responses API with store=True, or a Foundry agent owning the thread). With Chat Completions, OpenRouter, Ollama or local AgentSession history it is never set, and AgentSession.session_id is not exported, so every agent.run() becomes its own trace:<id> session.

Wrap every request/turn so all spans of that turn start inside conversation(<id>):

async def handle_message(session: AgentSession, text: str) -> str:
    with conversation(session.session_id):
        response = await agent.run(text, session=session)
    return response.text

• Id: the app's chat/thread id, or AgentSession.session_id (create sessions with agent.create_session(session_id=chat_id) when the app has an id). Must be stable across all turns of one conversation and differ between conversations.
• Streaming: the whole async for update in agent.run(..., stream=True) loop goes inside the with.
• Approval resumes (request.to_function_approval_response(...) passed back to agent.run) and workflow runs go inside the same with. 1.19 logs a WARN "Ignored an approval response ... did not match" on each resume even though the tool runs; ignore it if the execute_tool span is there once.
• Workflow executors (fan-out included) inherit the id.
• .NET: ConversationIdProcessor.Current.Value = chatId; in the request handler before RunAsync/RunStreamingAsync.

Step 4: Content

• Python: enable_sensitive_data=True (or ENABLE_SENSITIVE_DATA=true). .NET: EnableSensitiveData = true (or OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true).
• If the process sets OTEL_SEMCONV_STABILITY_OPT_IN, it must include gen_ai_latest_experimental (e.g. http,gen_ai_latest_experimental); otherwise MAF moves content off the spans into log events Maple doesn't read.
• enable_message_events=False: otherwise every message is also exported as OTLP log records (duplicate payload).
• gen_ai.tool.definitions (each tool's JSON schema) is sent on every invoke_agent span even with sensitive data off.
• If the user wants no prompt content stored, leave sensitive data off and tell them the transcript and tool arguments/results will be empty.

Step 5: Tools, errors, sub-agents

• Give every Agent a distinct name (Maple lanes key on gen_ai.agent.name; unnamed agents get a UUID).
• Tool failures: raising from the tool function is enough; MAF sets ERROR + error.type on execute_tool. Do not catch and return an error string from the tool body (that hides the failure).
• Python MAF also logs each tool failure as an ERROR and a WARN log record; configure_otel_providers() exports them as OTLP logs next to the span.
• Approval-gated tools (@tool(approval_mode="always_require")): the resume is a new agent.run() and trace; it joins the session only inside conversation(). A rejected call emits no execute_tool span; the rejection only appears in the next chat span's input messages, and only with sensitive data on.
• Workflows record fan-in as span links, so parallel executors are siblings under workflow.run.
• Sub-agents: prefer worker.as_tool() in the orchestrator's tools=[...], or MAF workflows/orchestrations. Do not also instrument the provider SDK (e.g. OpenInference/OpenLLMetry OpenAI instrumentors): that double-counts every call.

Step 6: Flush

Scripts, CLIs, notebooks, tests, serverless:

from opentelemetry import _logs, metrics, trace

try:
    asyncio.run(main())
finally:
    for provider in (trace.get_tracer_provider(), metrics.get_meter_provider(), _logs.get_logger_provider()):
        provider.shutdown()

Serverless handler kept warm: trace.get_tracer_provider().force_flush() before returning. Long-running servers: nothing per request. .NET: using var tracerProvider disposes at exit; hosted apps flush on graceful shutdown.

Semantic Kernel

Python (≥ 1.44.1; uv needs --prerelease=allow for its azure-ai-agents dependency):

pip install "semantic-kernel>=1.44.1" opentelemetry-sdk opentelemetry-exporter-otlp-proto-http

In a module imported before anything that imports semantic_kernel:

import logging
import os

os.environ["SEMANTICKERNEL_EXPERIMENTAL_GENAI_ENABLE_OTEL_DIAGNOSTICS"] = "true"
os.environ["SEMANTICKERNEL_EXPERIMENTAL_GENAI_ENABLE_OTEL_DIAGNOSTICS_SENSITIVE"] = "true"

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

from maple_tracing import ConversationIdProcessor

provider = TracerProvider(resource=Resource.create({"service.name": "<service-name>"}))
provider.add_span_processor(ConversationIdProcessor())
key = os.environ.get("MAPLE_INGEST_KEY")
if key:
    provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(
        endpoint="https://ingest.maple.dev/v1/traces",
        headers={"Authorization": f"Bearer {key}"},
    )))
else:
    # A missing key disables export; it never stops the app.
    logging.getLogger(__name__).warning("MAPLE_INGEST_KEY is not set; Maple telemetry export is disabled")
trace.set_tracer_provider(provider)

• The env vars are read once at import; set later, SK emits no GenAI spans and no warning.
• Wrap each turn in with conversation(thread.id): (ChatHistoryAgentThread().id is stable; SK never exports it).
• Call agents with positional messages: await agent.get_response(text, thread=thread), agent.invoke_stream(text, thread=thread). The keyword form messages= records empty input.
• Transcript comes only from invoke_agent spans (ChatCompletionAgent). Model-call content is Python logging, which Maple doesn't read; code that uses the kernel without an agent has no transcript. Tell the user.
• Orchestrations on InProcessRuntime: wrap in with conversation(id), tracer.start_as_current_span("<run name>"): and call runtime.start() inside it, or the run splits into many traces.
• Flush with provider.shutdown() in finally.
• Other SK differences to expect (not setup bugs): tool spans are execute_tool <Plugin>-<function>; failing tools get ERROR + error.type but no result attribute; finish reasons are Python enum names (FinishReason.STOP), so Maple's reply-length and refusal checks can't read them; temperature=0 is omitted from spans.
• .NET SK: same env vars (or AppContext switches Microsoft.SemanticKernel.Experimental.GenAI.EnableOTelDiagnostics[Sensitive]), AddSource("Microsoft.SemanticKernel*"), same ConversationIdProcessor.

Step 7: Verify

Run one real conversation (2-3 turns, one tool call; a second conversation if cheap). No scriptable entry point (server, UI, REPL only): write a small driver for this run (one conversation id, 2+ turns, one tool call, flush before exit). With a real key, open Agent Sessions (https://app.maple.dev/agent-sessions, EU app.eu.maple.dev) after ~1 minute, or with the Maple MCP list_agent_sessions with search=<conversation id> returns one row. Without Maple access (MAPLE_TEST, no MCP): the run must exit with no export errors on stderr (Failed to export, 401 lines) AND a local exporter (ConsoleSpanExporter added temporarily, or the endpoint pointed at a local collector) must show the spans below with gen_ai.conversation.id. Silence alone proves nothing (no spans is silent too). A 401 ingest_unauthorized / "Invalid ingest key" with a key you trust: keys are region-bound, so it likely belongs to the other region; try the other endpoint. Check:

• Spans arrive and the process exits cleanly (flush ran); service.name is yours, not agent_framework / unknown_service.
• Every span of every turn in one conversation has the same gen_ai.conversation.id; a second conversation has a different one. In Maple: one session per conversation, not trace:<id> sessions.
• Each turn: invoke_agent <name> root (.NET: invoke_agent <name>(<agent id>)) with chat <model> and execute_tool <tool> descendants in the same trace (streamed turn included).
• chat spans: gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens (also on the streamed turn), gen_ai.input.messages / gen_ai.output.messages as JSON arrays of {role, parts} (Python MAF; SK: on invoke_agent only).
• execute_tool: real tool name, gen_ai.tool.call.arguments and gen_ai.tool.call.result; a raising tool has ERROR status + error.type, successful ones don't.
• Sub-agents have distinct gen_ai.agent.names.
• No attribute contains the model API key or Bearer .
• Cost: none is emitted; Maple shows sessions as unpriced. Framework label: "Microsoft Agent Framework" / "Semantic Kernel".

Tokens and cost notes

• chat spans carry gen_ai.usage.input_tokens / output_tokens plus cache-read, cache-creation and reasoning buckets when the provider returns them. The OpenAI chat-completions client requests include_usage itself when streaming.
• invoke_agent repeats the sum of its own chat calls; Maple nets it out, so don't strip it.
• gen_ai.provider.name is the client type (openai even when pointed at OpenRouter or Ollama); server.address holds the real base URL.
• .NET streamed chat spans carry gen_ai.response.time_to_first_chunk (time to first token in Maple); Python MAF doesn't record it.

Do not

• Do not pass conversation_id as an agent/run option to label traces: it disables in-memory history and becomes previous_response_id on the Responses client.
• Do not pass a bare string as Message("user", text) contents; use [text].