ag2-telemetry

Telemetry — OpenTelemetry instrumentation

When to use

The user wants to:

See per-turn / per-call latency breakdowns
Attribute token usage across operations
Push traces to Jaeger, Grafana Tempo, Datadog, Honeycomb, Langfuse, etc.
Debug a slow agent end-to-end with structured spans rather than print statements

If they just want quick stdout debugging, point them at LoggingMiddleware instead (see ag2-middleware).

Installation

pip install "ag2[openai,tracing]"

Required. Run this install before delivering the code. If you cannot run commands, state the exact pip install command.

60-second recipe

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter

from ag2 import Agent
from ag2.config import OpenAIConfig
from ag2.middleware.builtin import TelemetryMiddleware

# 1. Configure OpenTelemetry
resource = Resource.create({"service.name": "ag2-quickstart"})
tracer_provider = TracerProvider(resource=resource)
tracer_provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(tracer_provider)

# 2. Wire the middleware
agent = Agent(
    "assistant",
    prompt="You are a helpful assistant.",
    config=OpenAIConfig(model="gpt-4o-mini"),
    middleware=[
        TelemetryMiddleware(
            tracer_provider=tracer_provider,
            agent_name="assistant",
        ),
    ],
)

# 3. Run — spans emit automatically
import asyncio
asyncio.run(agent.ask("What is the capital of France?"))

For production, swap ConsoleSpanExporter for OTLPSpanExporter (or your backend's exporter) and SimpleSpanProcessor for BatchSpanProcessor.

Span hierarchy

Each ask() produces a root span with children:

invoke_agent assistant
  ├── chat gpt-4o-mini              # LLM API call
  ├── execute_tool get_weather      # tool execution
  ├── chat gpt-4o-mini              # LLM call after tool result
  └── await_human_input assistant   # human-in-the-loop

Span types

Every span has an ag2.span.type attribute:

`ag2.span.type`	Operation name	Hook
`agent`	`invoke_agent`	`on_turn` — full turn
`llm`	`chat`	`on_llm_call` — each LLM call
`tool`	`execute_tool`	`on_tool_execution` — each tool
`human_input`	`await_human_input`	`on_human_input` — HITL

Semantic attributes (GenAI semconv)

Spans carry standard OpenTelemetry GenAI attributes:

Attribute	Spans	Description
`gen_ai.operation.name`	All	`invoke_agent` / `chat` / `execute_tool` / `await_human_input`
`gen_ai.agent.name`	agent, human_input	Agent name
`gen_ai.provider.name`	agent, llm	Auto-detected (`openai`, `anthropic`, …)
`gen_ai.request.model`	agent, llm	e.g. `gpt-4o-mini`
`gen_ai.response.model`	llm	Resolved from response
`gen_ai.response.finish_reasons`	llm	e.g. `["stop"]`, `["tool_calls"]`
`gen_ai.usage.input_tokens`	llm	Prompt tokens
`gen_ai.usage.output_tokens`	llm	Completion tokens
`gen_ai.usage.cache_creation_input_tokens`	llm	Prompt-cache writes (Anthropic)
`gen_ai.usage.cache_read_input_tokens`	llm	Prompt-cache reads (Anthropic, OpenAI, Gemini)
`gen_ai.tool.name`	tool	Tool function name
`gen_ai.tool.call.id`	tool	Tool call ID
`gen_ai.tool.type`	tool	Always `function`

Content capture (default ON)

By default, message content, tool args, and results are included on spans. Useful for debugging but can leak sensitive data:

TelemetryMiddleware(
    tracer_provider=tracer_provider,
    agent_name="assistant",
    capture_content=False,   # omit messages, tool args, results
)

When enabled, additional attributes appear:

Attribute	Span	Content
`gen_ai.input.messages`	llm	JSON request messages
`gen_ai.output.messages`	llm	JSON response messages
`gen_ai.tool.call.arguments`	tool	Tool args (JSON)
`gen_ai.tool.call.result`	tool	Tool result
`ag2.human_input.prompt`	human_input	Prompt shown to human
`ag2.human_input.response`	human_input	Human's response

For privacy-sensitive backends (or anywhere telemetry leaves your infra), set capture_content=False.

Constructor reference

Parameter	Type	Default	Description
`tracer_provider`	`TracerProvider \| None`	Global provider	OpenTelemetry TracerProvider
`capture_content`	`bool`	`True`	Include message/tool content in spans
`agent_name`	`str \| None`	`"unknown"`	Agent name for span attributes
`provider_name`	`str \| None`	`None`	Provider override (auto-detected if unset)
`model_name`	`str \| None`	`None`	Model override (auto-detected if unset)

Backend integration

TelemetryMiddleware uses standard OpenTelemetry, so any OTLP-compatible backend works:

Jaeger — OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")
Grafana Tempo — same OTLP exporter, point at the Tempo gateway
Langfuse, Honeycomb, Datadog — vendor-specific exporters; the agent-side setup is identical

For container-orchestrated stacks, this repo includes a tracing/ directory with Docker-Compose for otel-collector + Grafana Tempo.

Going deeper

website/docs/user-guide/telemetry.mdx — full attribute table, configuration, example.
tracing/ — Docker setup for local otel-collector + Tempo + Grafana.
For sibling middleware (logging, retry, history limits), see ag2-middleware.

Common pitfalls

SimpleSpanProcessor + ConsoleSpanExporter in production — synchronous, blocks every span emit. Use BatchSpanProcessor and a real exporter (OTLP / Jaeger / vendor) outside of dev.
Leaking content into telemetry — capture_content=True is the default. For privacy-sensitive prompts (PII, credentials), set capture_content=False and audit what your backend retains.
Forgetting trace.set_tracer_provider(...) — without it, tracer_provider you pass to the middleware is fine, but third-party libraries that auto-instrument may use a different provider.
Token usage missing — gen_ai.usage.* requires the provider client to surface usage in the response. Streaming providers may emit usage only at the end; if you don't see them, check the provider's response shape.
Span hierarchy doesn't show parent-child — your exporter or backend may need the OTLP/HTTP path enabled, not just OTLP/gRPC. Check both.
Comparing to AG2 Classic tracing docs — the semantic-attribute format is the same; only the agent instrumentation method differs (AG2 Classic uses instrument_agent() / instrument_llm_wrapper() / instrument_pattern(); AG2 uses TelemetryMiddleware).