SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsmaple-agent-tracing-spring-ai

maple-agent-tracing-spring-ai

Agents & models

View repository

Updated 10-16 00:05

About this skill

Maple agent tracing for Spring AI

Goal: every conversation with the Spring AI app shows up in Maple Agent Sessions as exactly one session, one turn per ChatClient call, with transcript, model calls, tool calls (failures marked), sub-agent lanes and tokens.

Mechanism: Spring AI's Micrometer Observations → micrometer-tracing-bridge-otel → OpenTelemetry SDK → OTLP/HTTP to Maple, all from spring-boot-starter-opentelemetry. Out of the box: sampling is 10%, prompts/replies never reach spans (log-prompt/log-completion only log to SLF4J), and thrown tool errors end the span OK. Steps 2-5 fix all three.

Step 0: Detect versions and existing setup

  1. Read pom.xml / build.gradle(.kts): Spring Boot version, spring-ai-bom version, model starter (spring-ai-starter-model-*). Target Spring AI 2.0.x (verified 2.0.1) on Boot 4.x (verified 4.1.1), Java 17+. Spring AI 1.1.x on Boot 3.5: see Step 2d.
  2. Grep for existing tracing: micrometer-tracing-bridge, spring-boot-starter-opentelemetry, opentelemetry-exporter-otlp, management.otlp, management.opentelemetry, management.tracing, -javaagent, opentelemetry-javaagent, OTEL_EXPORTER_OTLP, ObservationFilter, ObservationPredicate, ToolExecutionExceptionProcessor.
    • Existing Boot tracing to another backend: Boot has one OTLP span exporter. Ask the user whether to repoint it at Maple; to keep both, add a second exporter via an OtlpHttpSpanExporter bean only if they insist.
    • OTel Java agent attached (-javaagent:...opentelemetry-javaagent.jar): see Step 2c.
    • Existing ToolExecutionExceptionProcessor bean: patch it (Step 5) instead of adding the one in Step 3.
  3. Find every ChatClient call site (.prompt(, .call(), .stream()) and where the conversation/thread id lives in the request. Find every ChatClient.Builder (each role/sub-agent).

Step 1: Key and region

  • US: https://ingest.maple.dev. EU: https://ingest.eu.maple.dev.
  • Header: Authorization=Bearer <key>. Protocol: OTLP/HTTP protobuf (Boot's default transport).
  • Key in the user's prompt: use it. No key: use the literal MAPLE_TEST (ingest accepts and discards it) and tell the user to replace it with their key from Settings → Ingestion.
  • Private maple_sk_ keys never go in browser code.
  • Follow the repo's existing secret/env convention (${ENV_VAR} placeholders, profile files, Vault/Config Server). If there is none, inline in application.properties is acceptable because ingest keys are write-only.
  • A key from ${MAPLE_INGEST_KEY} must never stop the app or send an empty Bearer (opaque 401). Use the empty default ${MAPLE_INGEST_KEY:} (a bare ${MAPLE_INGEST_KEY} fails startup when unset) and turn export off in main when the key is missing, before SpringApplication.run (Step 2b shows it). That check reads the process environment and Boot does not read .env files, so export the variable where the JVM starts.

Step 2: Install and export

2a. Dependencies (Boot 4, Spring AI 2.0)

Keep the existing spring-ai-bom import and model starter. If there is no BOM yet, import org.springframework.ai:spring-ai-bom:2.0.1 (<type>pom</type><scope>import</scope> in dependencyManagement; Gradle implementation(platform("org.springframework.ai:spring-ai-bom:2.0.1"))). Add:

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-opentelemetry</artifactId>
</dependency>

Gradle: implementation("org.springframework.boot:spring-boot-starter-opentelemetry"). Actuator is not needed on Boot 4 (verified); keep it if the app already has it.

2b. Properties

Add to application.properties (or the YAML equivalent):

spring.application.name=<service name, e.g. support-agent>

management.opentelemetry.tracing.export.otlp.endpoint=https://ingest.maple.dev/v1/traces
management.opentelemetry.tracing.export.otlp.headers.Authorization=Bearer ${MAPLE_INGEST_KEY:}
management.tracing.sampling.probability=1.0
management.opentelemetry.resource-attributes.deployment.environment.name=<env>

management.otlp.metrics.export.url=https://ingest.maple.dev/v1/metrics
management.otlp.metrics.export.headers.Authorization=Bearer ${MAPLE_INGEST_KEY:}

maple.ai.capture-content=true

Then, in the application's main, disable export when the key is unset (merge with an existing main; keep its SpringApplication call):

public static void main(String[] args) {
	// A missing key disables export; it never stops the app.
	var mapleKey = System.getenv("MAPLE_INGEST_KEY");
	if (mapleKey == null || mapleKey.isEmpty()) {
		System.err.println("MAPLE_INGEST_KEY is not set; Maple telemetry export is disabled");
		System.setProperty("management.tracing.export.enabled", "false");
		System.setProperty("management.otlp.metrics.export.enabled", "false");
	}
	SpringApplication.run(Application.class, args);
}
  • The endpoint property takes the FULL URL including /v1/traces (Boot does not append it).
  • management.tracing.sampling.probability=1.0 is REQUIRED. Default is 0.1: 90% of turns silently missing.
  • The starter also exports metrics, to localhost:4318 by default. Either point them at Maple (above) or set management.otlp.metrics.export.enabled=false. Never leave the default.
  • Streamed tokens: Spring AI 2.0's OpenAI model requests usage on streams by default. If the app sets ANY spring.ai.openai.chat.stream-options.* property, also set spring.ai.openai.chat.stream-options.include-usage=true (once stream options exist, unset means false and streamed turns report no tokens).
  • Boot 4.1+ also maps OTEL_EXPORTER_OTLP_ENDPOINT (base URL, Boot appends /v1/traces), OTEL_EXPORTER_OTLP_HEADERS, OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES. Use them if the repo configures via env; keep the sampling property.
  • Do NOT set management.opentelemetry.tracing.limits.max-attribute-value-length: truncated JSON content no longer parses and Maple drops it.
  • Leave spring.ai.chat.observations.log-prompt/log-completion and spring.ai.chat.client.observations.* as they are. They only log; they never put content on spans.
  • @SpringBootTest disables tracing export; add @AutoConfigureTracing only if the user wants spans from tests.

2c. OTel Java agent present

The agent does not turn Micrometer Observations into spans, so Steps 2a-2b and 3 are still required. But agent + starter as-is is BROKEN: Boot's SDK and the agent don't share context, so every Spring AI span becomes its own trace (no hierarchy, model calls split from their session). Verified with agent 2.31.1. Either remove the agent for this service, or hand Micrometer the agent's OpenTelemetry (add ONLY while the agent is attached; without it GlobalOpenTelemetry.get() is a no-op and nothing is traced):

@Bean
io.opentelemetry.api.OpenTelemetry openTelemetry() {
	return io.opentelemetry.api.GlobalOpenTelemetry.get();
}

Then the agent exports everything: set OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev, OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer <key>, OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=<env> for the agent. Boot's management.opentelemetry.* export and sampling properties no longer apply (agent default sampler records everything). Also add -Dotel.instrumentation.openai-java.enabled=false as a precaution (no duplicate chat spans seen with Spring AI 2.0.1, but the agent ships OpenAI SDK instrumentation). HTTP client spans appear twice (Boot + agent); they are not AI spans and don't affect sessions.

2d. Spring AI 1.1 on Boot 3.5

  • Deps: io.micrometer:micrometer-tracing-bridge-otel + io.opentelemetry:opentelemetry-exporter-otlp + spring-boot-starter-actuator (required on Boot 3.5; no spring-boot-starter-opentelemetry).
  • Properties: management.otlp.tracing.endpoint=https://ingest.maple.dev/v1/traces, management.otlp.tracing.headers.Authorization=Bearer ...; sampling property unchanged. Stream usage: spring.ai.openai.chat.options.stream-usage=true.
  • Step 3 class: replace tools.jackson.databind.json.JsonMapper.shared().writeValueAsString(...) with a Jackson 2 com.fasterxml.jackson.databind.ObjectMapper (wrap the checked JsonProcessingException), and delete the two getToolCallId() lines (not in 1.1).
  • OpenAI model property: spring.ai.openai.chat.options.model on 1.1 (2.0: spring.ai.openai.chat.model).
  • Boot 4 still accepts the Boot 3 management.otlp.tracing.* names but marks them deprecated.

Step 3: Add the configuration class

Create it in a package the app's @SpringBootApplication scans (same package or below). Copy verbatim; change only the package:

package com.example.agent;

import java.util.ArrayList;
import java.util.List;
import java.util.Map;

import io.micrometer.common.KeyValue;
import io.micrometer.observation.Observation;
import io.micrometer.observation.ObservationFilter;
import io.micrometer.observation.ObservationRegistry;
import tools.jackson.databind.json.JsonMapper;

import org.springframework.ai.chat.client.observation.ChatClientObservationContext;
import org.springframework.ai.chat.messages.AssistantMessage;
import org.springframework.ai.chat.messages.Message;
import org.springframework.ai.chat.messages.ToolResponseMessage;
import org.springframework.ai.chat.observation.ChatModelObservationContext;
import org.springframework.ai.tool.execution.DefaultToolExecutionExceptionProcessor;
import org.springframework.ai.tool.execution.ToolExecutionExceptionProcessor;
import org.springframework.ai.tool.observation.ToolCallingObservationContext;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

/** Adds the gen_ai.* attributes Maple reads on top of Spring AI's own observations. */
@Configuration(proxyBeanMethods = false)
public class MapleAiObservationConfig {

	/** Agent name for a ChatClient: `.defaultAdvisors(a -> a.param(AGENT_NAME, "support_agent"))`. */
	public static final String AGENT_NAME = "gen_ai.agent.name";

	@Bean
	ObservationFilter mapleGenAiAttributes(@Value("${maple.ai.capture-content:false}") boolean captureContent) {
		return context -> {
			if (context instanceof ChatClientObservationContext client) {
				// One ChatClient call is one agent turn; Spring AI labels it "framework".
				client.addLowCardinalityKeyValue(KeyValue.of("gen_ai.operation.name", "invoke_agent"));
				if (client.getRequest().context().get(AGENT_NAME) instanceof String agent) {
					client.addLowCardinalityKeyValue(KeyValue.of("gen_ai.agent.name", agent));
				}
			}
			else if (context instanceof ToolCallingObservationContext tool) {
				tool.addLowCardinalityKeyValue(KeyValue.of("gen_ai.tool.name", tool.getToolDefinition().name()));
				if (tool.getToolCallId() != null) {
					tool.addHighCardinalityKeyValue(KeyValue.of("gen_ai.tool.call.id", tool.getToolCallId()));
				}
				if (captureContent) {
					tool.addHighCardinalityKeyValue(KeyValue.of("gen_ai.tool.call.arguments", tool.getToolCallArguments()));
					if (tool.getToolCallResult() != null) {
						tool.addHighCardinalityKeyValue(KeyValue.of("gen_ai.tool.call.result", tool.getToolCallResult()));
					}
				}
			}
			else if (captureContent && context instanceof ChatModelObservationContext chat) {
				chat.addHighCardinalityKeyValue(KeyValue.of("gen_ai.input.messages",
						JsonMapper.shared().writeValueAsString(chat.getRequest().getInstructions().stream().map(MapleAiObservationConfig::message).toList())));
				if (chat.getResponse() != null) {
					chat.addHighCardinalityKeyValue(KeyValue.of("gen_ai.output.messages",
							JsonMapper.shared().writeValueAsString(chat.getResponse().getResults().stream().map(g -> message(g.getOutput())).toList())));
				}
			}
			return context;
		};
	}

	// Spring AI hands a failing tool's message back to the model and ends the tool span OK.
	// Mark the span failed first, then keep the default behaviour.
	@Bean
	ToolExecutionExceptionProcessor toolExecutionExceptionProcessor(ObservationRegistry registry) {
		ToolExecutionExceptionProcessor fallback = DefaultToolExecutionExceptionProcessor.builder().build();
		return exception -> {
			Observation toolCall = registry.getCurrentObservation();
			if (toolCall != null) {
				toolCall.error(exception);
			}
			return fallback.process(exception);
		};
	}

	/** One message in the OpenTelemetry GenAI shape: {role, parts: [...]}. */
	private static Map<String, Object> message(Message message) {
		List<Map<String, Object>> parts = new ArrayList<>();
		if (message instanceof ToolResponseMessage toolResponse) {
			toolResponse.getResponses().forEach(r -> parts.add(
					Map.of("type", "tool_call_response", "id", r.id(), "response", r.responseData())));
		}
		else {
			if (message.getText() != null && !message.getText().isEmpty()) {
				parts.add(Map.of("type", "text", "content", message.getText()));
			}
			if (message instanceof AssistantMessage assistant) {
				assistant.getToolCalls().forEach(call -> parts.add(
						Map.of("type", "tool_call", "id", call.id(), "name", call.name(), "arguments", call.arguments())));
			}
		}
		return Map.of("role", message.getMessageType().getValue(), "parts", parts);
	}

}

Boot applies ObservationFilter beans to the registry automatically; nothing else to register. spring.ai.tools.observations.include-content writes spring.ai.tool.call.arguments/result, which Maple does not read; the filter's gen_ai.tool.call.* keys are the ones read.

Content notes: every chat span carries the whole conversation so far, so spans grow with long chats (don't cap them; see 2b). With maple.ai.capture-content=false no message/tool content leaves the process, except a failed tool's exception message and stack trace (toolCall.error(exception), Step 3); if those can hold user data, pass error() an exception without them. To redact content instead, mask values inside message(...). The conversation id and agent names are sent regardless: keep personal data out of them.

Kotlin project: translate one to one (e.g. ObservationFilter { context -> ...; context }), same beans, same keys.

Do NOT drop advisor observations with an ObservationPredicate: on .stream() calls Spring AI takes the model span's parent from the Reactor context, which then holds the skipped (no-op) advisor observation, so the streamed chat span becomes its own trace outside the session (verified).

Step 4: Session id (one conversation = one session)

Maple reads spring.ai.chat.client.conversation.id for Spring AI, on the chat_client span. Spring AI sets it from the ChatMemory.CONVERSATION_ID advisor param.

  • On EVERY top-level call: chatClient.prompt().user(msg).advisors(a -> a.param(ChatMemory.CONVERSATION_ID, conversationId)).... Use the id the app already stores for the chat/thread.
  • If the app already passes this param for MessageChatMemoryAdvisor/PromptChatMemoryAdvisor/VectorStoreChatMemoryAdvisor, nothing to add. If it has no chat memory, add the param anyway; it works without a memory advisor.
  • Never set the conversation id via defaultAdvisors(...) on a shared builder/client (constant id = all users in one session). Never mint a UUID per request.
  • Sub-agent ChatClient calls inside tools need no id: Maple groups the whole trace by any span carrying it. Don't pass a different id to sub-agents.
  • Give every ChatClient an agent name: builder.defaultAdvisors(a -> a.param(MapleAiObservationConfig.AGENT_NAME, "<snake_case_role>")).

Step 5: Tools, errors, sub-agents

  • Tool spans are execute_tool <name>; Step 3 adds gen_ai.tool.name, gen_ai.tool.call.id and (with content on) arguments/result.
  • Failures: Step 3's ToolExecutionExceptionProcessor marks the tool span ERROR (exception message + exception event) and still returns the message to the model. If the app has its own processor bean, do not add a second one: insert Observation current = registry.getCurrentObservation(); if (current != null) current.error(exception); at the top of its process(...). If the app relied on spring.ai.tools.throw-exception-on-error=true, build the fallback with .alwaysThrow(true) (the property no longer applies once the bean is replaced).
  • Tools that return error strings instead of throwing count as success. Point it out; don't change behavior unasked.
  • Sub-agents: one ChatClient per role, exposed to the orchestrator as a @Tool(name = "<role>", description = ...) method that calls it. Build each from builder.clone() with its own AGENT_NAME param. Maple shows execute_tool <role> → worker chat_client as a delegation lane.
  • Own thread pools / CompletableFuture fan-out: propagate the observation to worker threads (Micrometer context-propagation: ContextSnapshot, a wrapped executor, or build the worker observation with .parentObservation(parent)). Otherwise each worker starts a new trace and becomes its own trace:<id> session.
  • Spring AI runs the tool calls of one model response sequentially on the calling thread, so context flows without help unless the app fans out itself.
  • A ChatClient without an agent name gets no lane; its model and tool calls are drawn in the caller's lane.
  • Spring AI 2.0 has no native tool-approval/HITL mechanism; don't invent one.

Sub-agent example:

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.tool.annotation.Tool;
import org.springframework.stereotype.Component;

@Component
public class Workers {

	private final ChatClient weather;

	public Workers(ChatClient.Builder builder, WeatherTools weatherTools) {
		this.weather = builder.clone()
			.defaultSystem("You answer weather questions using your tools.")
			.defaultTools(weatherTools)
			.defaultAdvisors(a -> a.param(MapleAiObservationConfig.AGENT_NAME, "weather_worker"))
			.build();
	}

	@Tool(name = "weather_worker", description = "Ask the weather specialist about a city")
	public String weatherWorker(String task) {
		return weather.prompt().user(task).call().content();
	}

}

Step 5b: Tokens and cost (no action needed, explain if asked)

  • Spring AI records the provider as gen_ai.system, derived from the client class, not the model: a Claude model behind OpenRouter via the OpenAI starter is labeled openai.
  • No cost attribute is emitted; sessions show as unpriced (Maple never prices tokens).

Step 6: Flush

  • Web app: nothing. Boot shuts down the SdkTracerProvider on context close, which flushes.
  • CommandLineRunner / batch: exit explicitly with System.exit(SpringApplication.exit(SpringApplication.run(App.class, args))) in main. If main just returns, the OpenAI starter's HTTP client keeps non-daemon threads alive ~60 s; the context (and the final span flush) only closes after that (verified). Runtime.halt() / SIGKILL lose the last batch.
  • Serverless (Spring Cloud Function on Lambda etc.): inject io.opentelemetry.sdk.trace.SdkTracerProvider and call tracerProvider.forceFlush().join(10, TimeUnit.SECONDS) at the end of EVERY invocation; never shut it down.
  • Batch delay is 5 s; wait before checking Maple.

Step 7: Verify

If the app has no scriptable entry point (web or UI only), write a small CommandLineRunner driver for this run: one conversation id, 2+ turns, at least one tool call, exit as in Step 6.

Run one real conversation: 2-3 turns with the same conversation id including one tool call (one streamed turn if the app streams), plus a second conversation with a different id. Exit cleanly. Wait ~1 minute. In Maple Agent Sessions, filtered by the service name (or via the Maple MCP: list_agent_sessions with search=<conversation id> returns one row; open it with get_agent_session), check:

  • Exactly one session per conversation id; none named trace:<id> (that means a top-level call lacked the CONVERSATION_ID param, or a sub-agent ran on a thread without context).
  • The two conversations are two different sessions.
  • Framework shows Spring AI.
  • Every turn arrived (count = number of top-level ChatClient calls). Missing turns = sampling property not applied.
  • Spans: spring_ai chat_client (agent, with your agent name), chat <model>, execute_tool <tool>.
  • LLM call count = number of chat <model> spans (not doubled); tool call count = number of real tool invocations.
  • Transcript shows user messages, replies, tool calls with arguments and results (else maple.ai.capture-content isn't true or the class isn't scanned).
  • Input and output tokens on every model call, including the streamed one.
  • A tool that threw is counted as failed, with its message; successful tools are not.
  • Sub-agents appear as lanes under their agent names.
  • Cost shows as unpriced (Spring AI emits no cost; expected).
  • App logs have no Failed to publish metrics / OTLP export errors, no 401. A 401 (ingest_unauthorized / "Invalid ingest key") with a key you trust usually means the key belongs to the other region (keys are region-bound): try the other endpoint. Otherwise check the header property reads ...headers.Authorization=Bearer <key>.

Without Maple access: the run logs no OTLP export errors or 401s AND a temporary LoggingSpanExporter bean (io.opentelemetry:opentelemetry-exporter-logging) logs the chat_client spans with spring.ai.chat.client.conversation.id. Silence alone proves nothing: no spans also looks silent.

Do not

  • Do not stamp maple_ai.session.id on Spring AI spans; the conversation id param is the supported path.