maple-agent-tracing-eve

Maple agent tracing: eve

Goal

One eve session (wrun_...) = one Maple Agent Session. One user message = one turn = one trace rooted at invoke_agent <agent>, with agent.step spans holding a chat <model> span (model, provider, tokens, messages) and execute_tool <tool> spans (arguments, result, failures). Subagents nest inside the caller's trace and show as their own lanes.

eve owns the tracer provider and writes gen_ai.conversation.id (the root session id) on every agent span. You add a destination; you never start an SDK and never pass a conversation id.

Known gaps (tell the user, don't try to fix):

• Outside eve dev, message content is exported only for public conversations (Step 4).
• Cost shows as unpriced with a model called directly at its provider. With Vercel AI Gateway, eve writes gen_ai.usage.cost on the agent.step span; Maple counts it in the session total (the per-model table has tokens, not cost).
• eve does not log failed exports (Step 6).

Step 0: Detect

• eve version in package.json / lockfile.
  • >= 0.62: this skill. Exercised on 0.62.0, 0.70.3, 0.72.1, 0.77.0 and 0.78.0 with the Step 2/3 files unchanged; every version produced complete sessions in Maple.
  • 0.62 to 0.71: each tool call is an agent.action span wrapping execute_tool. Maple shows the wrapper as a structure row and counts one tool call. Expected.
  • < 0.62: telemetry lives in agent/instrumentation.ts (defineInstrumentation({ setup }) calling registerOTel), and eve ignores agent/instrumentation/ without an error: the Step 2 files export nothing (tested on 0.61.1). With experimental: { instrumentationProviders: true } in agent.ts they export, but the exportPolicy is ignored and every workflow span goes out. Don't use the flag: upgrade (npm install eve@latest), delete agent/instrumentation.ts (until then eve build fails with Found removed instrumentation file), remove the flag, then do Steps 2-3. Move every other exporter from the old setup into its own otelIntegration({ traceExporter }) file; serviceName becomes otel({ resource }), functionId becomes otel({ functionId }), events["step.started"] runtime context becomes otelIntegration({ runtimeContext }). See https://eve.dev/docs/observability/instrumentation-migration.
  • Tell the user: from 0.35 to 0.61 the old file recorded no content unless recordInputs/recordOutputs were set (before 0.35 it recorded content by default); after the upgrade, content is recorded during eve dev and for public conversations (Step 4). The first eve dev after the upgrade deletes the local .eve/.workflow-data (incompatible storage layout), so open local dev sessions are lost.
• Node.js >= 24 at runtime; eve enforces it. bun or pnpm as the package manager is fine.
• Layout: one agent in agent/, or a workspace with agents/<name>/agent/. In a workspace add the files to each member's agent/instrumentation/ (not exercised; run Step 7 per member).
• Existing files in agent/instrumentation/:
  • An otel() file: add the resource attributes from Step 3 to it. Two otel() files are a boot error.
  • Other otelIntegration() files (Honeycomb, Braintrust, Datadog...): keep them. Maple is one more file; each destination has its own exportPolicy.
  • agent-runs.ts, local.ts: leave them.
• Search for NodeSDK, registerOTel(, registerTelemetry(, @ai-sdk/otel, NodeTracerProvider. eve replaces all of them with otelIntegration(); don't add one, and tell the user if one exists.
• eve depends on ai. Use this skill, not maple-agent-tracing-vercel-ai-sdk.
• agent/skills/maple-agent-tracing-eve/ exists (an npx skills add run in an eve project installs into the eve agent's own skills, and skills-lock.json records it): it would ship with the agent as a runtime skill. Tell the user and remove both the directory and its skills-lock.json entry once setup is done, unless they want it there.
• Models: a string id in agent.ts (model: "openai/gpt-…") runs through Vercel AI Gateway; a provider object (openai() from eve/models/openai, createOpenRouter(...), anthropic()) calls the provider directly. This decides the cost line in Step 7.

Step 1: Key and region

• US https://ingest.maple.dev/v1/traces, EU https://ingest.eu.maple.dev/v1/traces. The exporter takes the full URL. Header authorization: Bearer <key>.
• Key in the user's prompt: use it. No key: use the literal MAPLE_TEST (ingest accepts and discards it) and tell the user to replace it with their key from Settings → Ingestion.
• Private maple_sk_ keys never go in browser code. Ingest keys are write-only.
• eve loads .env.local for eve dev and eve start: put MAPLE_INGEST_KEY=<key> there (check it is gitignored). Deployments read the host's environment variables.
• Instrumentation files run at server start, not at build: eve build works without the key.
• Never let an unset variable become Bearer undefined. When the key is missing, log one warning and export disableInstrumentation() so the agent runs normally (Step 2 does this).
• A 401 ingest_unauthorized with a key you trust usually means the key belongs to the other region: try the other endpoint.

Step 2: Install and add the destination

npm install @vercel/otel

Use the repo's package manager. Create agent/instrumentation/maple.ts:

import { OTLPHttpProtoTraceExporter } from "@vercel/otel"
import { disableInstrumentation } from "eve/instrumentation"
import { otelIntegration } from "eve/instrumentation/otel"

// A missing key disables export; it never stops the agent.
const mapleKey = process.env.MAPLE_INGEST_KEY
if (!mapleKey) console.warn("MAPLE_INGEST_KEY is not set; Maple telemetry export is disabled")

export default mapleKey
	? otelIntegration({
			traceExporter: new OTLPHttpProtoTraceExporter({
				url: "https://ingest.maple.dev/v1/traces",
				headers: { authorization: `Bearer ${mapleKey}` },
			}),
			// Keep the spans of a conversation; drop the workflow engine's own
			exportPolicy: { span: ({ attributes }) => ({ emit: "gen_ai.conversation.id" in attributes }) },
		})
	: disableInstrumentation()

• The file name is free; it must default-export the integration.
• eve wraps traceExporter in its own batch processor (5 s delay, flushes on shutdown). Don't wrap it in a BatchSpanProcessor yourself.
• The exportPolicy is required in practice. eve's second scope, workflow, is the durable-execution engine: workflow.*, world.*, step.*, hook.resume, queue.publish, including one workflow.stream.flush span per streamed chunk. In two test runs it was 1,031 of 1,060 and 1,794 of 1,829 spans, all in traces of their own with no gen_ai.conversation.id.
• The policy decides at span start. Every eve agent span has gen_ai.conversation.id at start. Per eve's docs the id is also on third-party instrumentation spans created during model, tool and memory execution; spans outside those (for example agent.channel.request with traceChannelRequests: true) are dropped by this policy.
• The user wants the workflow spans too: remove exportPolicy and tell them the volume.

Step 3: Service name and environment

Without settings, service.name is the package.json name. Create agent/instrumentation/otel.ts, or extend the existing otel() file:

import { otel } from "eve/instrumentation/otel"

export default otel({
	resource: {
		"service.name": "support-agent",
		"deployment.environment.name": process.env.VERCEL_ENV ?? "development",
	},
})

• Use the app's own environment variable for the environment (VERCEL_ENV on Vercel).
• gen_ai.agent.name is the package.json name for the root agent and the directory name for a declared subagent (agent/subagents/<name>/). They are the lane names in Maple.

Step 4: Content and audience

eve attaches content to spans only when the conversation's audience allows it. The channel assigns the audience once, when the session starts:

Runs as	Audience	Content exported
eve dev	any	yes
preview / production (eve start, Vercel)	public	yes
preview / production	private / unknown	no

• Without content the session still has turns, model calls, tools, tokens and failures. Missing: gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions, gen_ai.tool.definitions, gen_ai.tool.call.arguments, gen_ai.tool.call.result, the session title and the turn labels.
• Defaults: HTTP channel (eveChannel): authenticated user/service/runtime principals are private, anonymous callers unknown. Slack: public channels public; DMs, group DMs and private channels private.
• otel({ tracePolicy }) cannot widen this. Tested on eve start with a private caller and a policy returning { emit: true, recordInputs: true, recordOutputs: true }: the spans say agent.trace.content.input=true and still carry no content. The policy only narrows.
• The lever is the channel. HTTP channel: eveChannel({ auth: [...], audience: "public" }) in agent/channels/eve.ts. The file replaces the whole default channel, so keep the auth list the project already uses. audience also takes a function; a custom channel has an audience() hook. See https://eve.dev/docs/channels/overview#audience.
• Ask the user before setting audience: "public". It is a data decision: it applies to every trace destination, including Vercel Agent Runs, not only Maple. Don't set it on your own.
• To keep content out of Maple while other destinations keep it, add a second policy to the Maple destination:

exportPolicy: [
	{ span: ({ attributes }) => ({ emit: "gen_ai.conversation.id" in attributes }) },
	{ span: () => ({ redact: true, inputs: true, outputs: true }) },
],

• Each content attribute is capped at 32,768 characters by eve.

Step 5: Tools, subagents, approvals

• A tool's span name and gen_ai.tool.name are its file name under agent/tools/.
• Failures must throw. Thrown: span status ERROR with the message, error.type=ACTION_RESULT_FAILED, an exception event; Maple counts the tool call as failed. A returned { error } counts as success.
• Declared subagents: the call is execute_tool <subagent> and returns a task receipt within ~0.1 s; the child's invoke_agent <subagent> nests under it in the same trace. In Maple the lane's result is the receipt text ("Started task ..."), and the real result arrives through the framework tool eve__task_wait. Expected.
• Framework tools (eve__task_wait, bash, read_file...) are tool calls like any other.
• Approvals (approval: always() and friends): eve adds an agent.approval span under the tool call. The turn stays open while it waits, so the wait counts toward the turn's and the tool call's duration. A denied call is a failed tool call with error.type=TOOL_EXECUTION_DENIED, which raises Maple's "Tool errors" check. Expected.
• Tokens: invoke_agent spans repeat the turn total and agent.step repeats the call's usage under agent.usage.*. Maple counts the chat spans only; totals are not doubled.

Step 6: Flush and silent failures

• Add no flush code. On eve start, a SIGTERM right after a turn still delivered every span. Vercel deployments were not exercised.
• eve does not report a failed export, also not with EVE_LOG_LEVEL=warn. A wrong key or region looks exactly like no traffic. Test the key directly:

curl -s -w "%{http_code}\n" -X POST https://ingest.maple.dev/v1/traces \
  -H "Authorization: Bearer <key>" \
  -H "Content-Type: application/json" -d '{"resourceSpans":[]}'

200 = valid. 401 prints the reason, including the right endpoint when the key is from the other region.

Step 7: Verify

Run one real conversation: eve dev (or eve dev --no-ui plus a script on eve/client), 2+ messages in one session, one of them calling a tool; plus a second session. If the project has subagents or an approval-gated tool, exercise each once. Then check (Maple MCP list_agent_sessions filtered by service, get_agent_session_transcript, or the Agent Sessions page, ~30 s after the run):

• The startup log has no MAPLE_INGEST_KEY is not set.
• One session per eve session, id wrun_...; the second session is separate; no trace:<id> sessions.
• One turn per user message, labeled with the message. A turn paused for an approval is still one turn.
• Transcript shows user messages, the system instructions, assistant replies and tool calls with arguments and results.
• Every chat <model> span has model, provider, input and output tokens. LLM calls = number of chat spans.
• A throwing tool is failed with its message; successful tools are not.
• Subagents show as lanes named after their directory, inside the calling turn.
• No span named workflow.*, world.*, step.*, hook.* or queue.* for the service (Maple MCP search_traces, or the Traces page).
• Framework shows eve, not Unidentified.
• Cost: present for AI Gateway models, unpriced for direct provider models (see Known gaps).
• Then start the production build once (eve build && eve start) and send one message. Expect the session without content unless the audience is public; tell the user which it is.

Without Maple access: the curl check in Step 6 returns 200, the startup log is clean, and a conversation ran. That shows delivery is possible, not that spans arrived; say so.

Do not

• Do not start NodeSDK, registerOTel or registerTelemetry, and do not add @ai-sdk/otel, OpenLLMetry or OpenInference: eve already traces every model call.
• Do not stamp gen_ai.conversation.id or maple_ai.session.id yourself.
• Do not add a second otel() file.
• Do not set audience: "public" without the user's decision.