maple-agent-tracing-genkit

Maple agent tracing: Genkit

Goal

One conversation = one Maple Agent Session, one turn per flow run, with the transcript, every model call (model, tokens) and every tool call (name, args, result, failures).

Span tree per flow run:

supportChat            genkit:metadata:subtype=flow, genkit:isRoot=true  -> invoke_agent
└── generate           genkit:type=util (one nested generate per tool-loop step, left unmapped)
    ├── googleai/gemini-2.5-flash   subtype=model  -> chat
    ├── getWeather                  subtype=tool   -> execute_tool
    └── generate
        └── googleai/gemini-2.5-flash   subtype=model  -> chat

Beta agents (ai.defineAgent() / defineCustomAgent / definePromptAgent from genkit/beta): root span has subtype agent and genkit:metadata:agent:sessionId (Genkit's session id), which the processor uses as the conversation id. Under it: runTurn-<n> (flowStep), render (promptTemplate), generate, model, tool spans. One trace per chat.send().

Known gaps (tell the user, don't try to fix): cost shows as unpriced; gen_ai.provider.name is the Genkit plugin prefix (googleai, vertexai, openai, anthropic...) rather than the semconv value (gcp.gemini...), which is only a label in Maple; media parts are left out of transcripts; execute_tool spans have no gen_ai.tool.call.id (Genkit doesn't put the call ref on the tool span; the transcript still pairs calls and results through the ids in the model messages when the model plugin sets ref).

Step 0: Detect

• genkit version in package.json / lockfile: need >= 1.22 (disableGenkitOTelInitialization was added in 1.22). Older: upgrade Genkit first. Node.js >= 20.
• Go or Python Genkit: stop; this skill covers TypeScript/JavaScript only. Tell the user.
• Existing OpenTelemetry: search for NodeSDK, NodeTracerProvider, registerOTel, @vercel/otel, Sentry.init, enableTelemetry(, enableFirebaseTelemetry(, enableGoogleCloudTelemetry(, ENABLE_FIREBASE_MONITORING.
  • An SDK/provider already exists: reuse it. Add GenkitForMaple and one Maple exporting processor to it. Never start a second SDK.
  • enableTelemetry({...}) from genkit/tracing with custom processors: remove it. Genkit's enableTelemetry builds its own bundled @opentelemetry/sdk-node 0.52 / sdk-trace-base 1.25, and current (2.x) exporters crash inside it (Cannot read properties of undefined (reading 'name') on instrumentationScope). Don't pass Maple's exporter there.
  • enableFirebaseTelemetry() / enableGoogleCloudTelemetry() / ENABLE_FIREBASE_MONITORING=true: disableGenkitOTelInitialization() makes these no-ops (they go through enableTelemetry). Ask the user whether Google Cloud trace export may stop. If they need both, stop and tell them; don't wire two SDKs.
• Find every flow (ai.defineFlow(), direct ai.generate( / ai.generateStream( / prompt( call site, beta agents (defineAgent(, .chat(), and how the app deploys (plain Node server, Express startFlowServer/expressHandler, Next.js @genkit-ai/next, Cloud Functions for Firebase onCallGenkit, Cloud Run). Find each conversation's id (chat id, thread id, session row).

Step 1: Key and region

• US endpoint https://ingest.maple.dev, EU endpoint https://ingest.eu.maple.dev. Header Authorization=Bearer <key>.
• Key in the user's prompt: use it. No key: use the literal MAPLE_TEST (ingest accepts and discards it) and tell the user to replace it with their key from Settings → Ingestion.
• Private maple_sk_ keys never go in browser code. Ingest keys are write-only.
• Follow the repo's existing secret/env convention (.env, Firebase defineSecret, Secret Manager). If there is none, inlining the ingest key is acceptable.
• A 401 ingest_unauthorized / "Invalid ingest key" with a key you trust usually means the key belongs to the other region (keys are region-bound): try the other endpoint.

Step 2: Install

npm install genkit @opentelemetry/sdk-node @opentelemetry/sdk-trace-base @opentelemetry/exporter-trace-otlp-proto

Use the repo's package manager. @opentelemetry/api arrives as a peer; add it explicitly only if the package manager doesn't install peers. Env (or the repo's equivalent):

OTEL_SERVICE_NAME=support-agent
OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production
OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <key>"

Inlining instead of env: new OTLPTraceExporter({ url: "https://ingest.maple.dev/v1/traces", headers: { authorization: "Bearer <key>" } }) (the full /v1/traces path is needed when passing url).

• The app loads .env (dotenv, --env-file): load it at the top of instrumentation.ts (import "dotenv/config" as its first line) or run with --env-file. The exporter reads the OTEL_* vars when it is constructed; otherwise it silently targets localhost:4318 with no key.
• Building the header from a variable (Bearer ${process.env.MAPLE_INGEST_KEY}): when it is unset, log one warning (MAPLE_INGEST_KEY is not set; Maple telemetry export is disabled) and skip the Maple exporter so the app runs normally. Never throw over the key, and never let it become Bearer undefined (opaque 401).

Step 3: The span processor

Create genkit-for-maple.ts next to the entry point, verbatim:

// genkit-for-maple.ts
import type { ReadableSpan, SpanProcessor } from "@opentelemetry/sdk-trace-base"

type Part = {
	text?: string
	reasoning?: string
	toolRequest?: { name: string; ref?: string; input?: unknown }
	toolResponse?: { name: string; ref?: string; output?: unknown }
}
type Message = { role: string; content: Part[] }

// Genkit message parts to OpenTelemetry GenAI parts. Media parts are left out.
function toPart({ text, reasoning, toolRequest, toolResponse }: Part) {
	if (text !== undefined) return { type: "text", content: text }
	if (reasoning !== undefined) return { type: "reasoning", content: reasoning }
	if (toolRequest) {
		return { type: "tool_call", id: toolRequest.ref, name: toolRequest.name, arguments: toolRequest.input }
	}
	if (toolResponse) return { type: "tool_call_response", id: toolResponse.ref, response: toolResponse.output }
	return undefined
}

const toParts = (content: Part[]) => content.map(toPart).filter((part) => part !== undefined)

function toMessages(messages: Message[]) {
	return messages.map((m) => ({ role: m.role === "model" ? "assistant" : m.role, parts: toParts(m.content) }))
}

/** Adds the gen_ai.* attributes Maple reads to Genkit's flow, model and tool spans. */
export class GenkitForMaple implements SpanProcessor {
	onStart() {}

	onEnd(span: ReadableSpan) {
		const attrs = span.attributes
		const json = (key: string) => {
			const value = attrs[key]
			return typeof value === "string" ? JSON.parse(value) : undefined
		}
		const name = String(attrs["genkit:name"])

		switch (attrs["genkit:metadata:subtype"]) {
			case "flow":
			case "agent": {
				Object.assign(attrs, {
					"gen_ai.operation.name": "invoke_agent",
					"gen_ai.agent.name": name,
				})
				// Set in your flow, or by Genkit for defineAgent() chats
				const conversationId = attrs["genkit:metadata:conversationId"] ?? attrs["genkit:metadata:agent:sessionId"]
				if (conversationId !== undefined) attrs["gen_ai.conversation.id"] = conversationId
				break
			}
			case "model": {
				const input = json("genkit:input")
				const output = json("genkit:output")
				const [provider, ...model] = name.split("/")
				const messages: Message[] = input?.messages ?? []
				const system = messages.filter((m) => m.role === "system").flatMap((m) => toParts(m.content))
				Object.assign(attrs, {
					"gen_ai.operation.name": "chat",
					"gen_ai.provider.name": provider,
					"gen_ai.request.model": model.join("/") || name,
					"gen_ai.input.messages": JSON.stringify(toMessages(messages.filter((m) => m.role !== "system"))),
				})
				if (system.length > 0) attrs["gen_ai.system_instructions"] = JSON.stringify(system)
				if (output?.message) {
					attrs["gen_ai.output.messages"] = JSON.stringify(
						toMessages([output.message]).map((m) => ({ ...m, finish_reason: output.finishReason })),
					)
				}
				if (output?.finishReason) attrs["gen_ai.response.finish_reasons"] = [output.finishReason]
				if (output?.usage?.inputTokens !== undefined) attrs["gen_ai.usage.input_tokens"] = output.usage.inputTokens
				if (output?.usage?.outputTokens !== undefined) attrs["gen_ai.usage.output_tokens"] = output.usage.outputTokens
				break
			}
			case "tool": {
				Object.assign(attrs, {
					"gen_ai.operation.name": "execute_tool",
					"gen_ai.tool.name": name,
					"gen_ai.tool.call.arguments": attrs["genkit:input"] ?? "{}",
				})
				const result = json("genkit:output")
				if (result !== undefined) {
					attrs["gen_ai.tool.call.result"] = typeof result === "string" ? result : JSON.stringify(result)
				}
				break
			}
		}
	}

	forceFlush() {
		return Promise.resolve()
	}

	shutdown() {
		return Promise.resolve()
	}
}

It maps:

Genkit span (genkit:metadata:subtype)	Adds
flow, agent	gen_ai.operation.name=invoke_agent, gen_ai.agent.name=<genkit:name>, gen_ai.conversation.id from genkit:metadata:conversationId or genkit:metadata:agent:sessionId
model	chat, gen_ai.provider.name (prefix before /), gen_ai.request.model (rest), gen_ai.input.messages / gen_ai.system_instructions from genkit:input.messages, gen_ai.output.messages + gen_ai.response.finish_reasons from genkit:output, gen_ai.usage.input_tokens / output_tokens from genkit:output.usage
tool	execute_tool, gen_ai.tool.name, gen_ai.tool.call.arguments (= genkit:input), gen_ai.tool.call.result (= genkit:output, string results as plain text)

Message conversion: role model → assistant; parts {text} → text, {reasoning} → reasoning, {toolRequest:{name,ref,input}} → tool_call, {toolResponse:{name,ref,output}} → tool_call_response; media, data, custom parts are dropped.

Notes:

• It mutates span.attributes in onEnd. That works because the exporting processor reads the same object; list GenkitForMaple BEFORE the exporting processor (required with SimpleSpanProcessor, which exports inside onEnd).
• Optional, only if the user asks for cache/reasoning token detail: output.usage.cachedContentTokens → gen_ai.usage.cache_read.input_tokens, output.usage.thoughtsTokens → gen_ai.usage.reasoning.output_tokens. Gemini models (googleai/, vertexai/) report outputTokens without thoughts: for them set output_tokens to outputTokens + thoughtsTokens. Keep input_tokens as Genkit reports it.
• Optional provider label mapping (googleai → gcp.gemini, vertexai → gcp.vertex_ai): cosmetic, skip unless asked.
• Leave genkit:* attributes in place; they are what the Genkit Developer UI and Google Cloud views read.
• Typecheck the file with the repo's tsc (it passes strict).

Step 4: Start OpenTelemetry

Create instrumentation.ts:

import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto"
import { NodeSDK } from "@opentelemetry/sdk-node"
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base"
import { disableGenkitOTelInitialization } from "genkit/tracing"
import { GenkitForMaple } from "./genkit-for-maple"

export const spanProcessor = new BatchSpanProcessor(new OTLPTraceExporter())
export const sdk = new NodeSDK({ spanProcessors: [new GenkitForMaple(), spanProcessor] })

if (process.env.GENKIT_ENV !== "dev") {
	disableGenkitOTelInitialization()
	sdk.start()
}

• import "./instrumentation" as the FIRST line of every entry point (server, worker, CLI, Cloud Function index). It must run before Genkit's first span: otherwise Genkit's own SDK registers the global tracer provider first and Maple's never receives spans. Adjust the import extension/style to the repo (ESM .js suffix, tsx, bundler).
• Why disableGenkitOTelInitialization(): without it Genkit lazily starts a second, bundled NodeSDK on its first span. With Maple's SDK started first that second one mostly loses the global registration race, but it's two SDKs; disable it.
• GENKIT_ENV=dev is set by genkit start (Developer UI). Genkit's own SDK then ships traces to the Developer UI's telemetry server; Maple's SDK can't feed it (Genkit's TraceServerExporter reads 1.x span fields). So in dev: Developer UI traces, nothing to Maple. If the user wants Maple in dev too, drop the if and tell them the Developer UI trace view goes empty.
• Existing provider/SDK: add new GenkitForMaple() and the Maple BatchSpanProcessor to its span processors (before any exporting processor for GenkitForMaple), keep disableGenkitOTelInitialization(), don't create NodeSDK.
• Next.js with @vercel/otel: pass spanProcessors: [new GenkitForMaple(), new BatchSpanProcessor(new OTLPTraceExporter({ url, headers }))] to registerOTel and call disableGenkitOTelInitialization() in register(). Untested; verify spans arrive.
• NodeSDK() with spanProcessors doesn't build exporters from OTEL_TRACES_EXPORTER; the explicit OTLPTraceExporter still reads OTEL_EXPORTER_OTLP_ENDPOINT / HEADERS. It also starts OTLP metric and log exporters from env (default otlp); set OTEL_METRICS_EXPORTER=none / OTEL_LOGS_EXPORTER=none if the repo doesn't want them.

Genkit starts a NEW TRACE for every root Genkit action (it passes root: true), even inside an HTTP server span. That's fine for Maple (one trace per turn). If the user wants flows nested under their HTTP spans, disableOTelRootSpanDetection() from genkit/tracing changes it; not needed for Agent Sessions.

Step 5: Conversation id (session)

Every turn must run inside a flow (or a beta agent). In each flow that handles a conversation, call at the start:

import { setCustomMetadataAttribute } from "genkit/tracing"

setCustomMetadataAttribute("conversationId", chatId)

• It writes to the CURRENT Genkit action, so call it directly in the flow body, not inside a tool or a nested ai.run() step. It throws outside any Genkit action.
• ai.generate() called outside a flow (e.g. straight from an Express route): wrap the call in a flow (ai.defineFlow) and call the flow from the route. Without a flow there is no invoke_agent span and no session id; each call becomes its own trace:<id> session.
• Id: stable per conversation, unique across conversations. Never a module constant, Date.now() per call or a per-process id. If the app has no id, create one when the conversation is created and persist it with it.
• Flows exposed via startFlowServer / expressHandler / onCallGenkit / appRoute: the client must send the chat id in the flow input; add it to inputSchema if missing.
• Beta agents (defineAgent etc.): nothing to add; genkit:metadata:agent:sessionId is used.
• The flow name is the agent name in Maple (gen_ai.agent.name). Give each distinct agent its own flow name.
• Sub-agents: a flow called from inside another flow or a tool also becomes an invoke_agent span named after that flow (expected to show as a sub-agent lane; not verified in Maple). Don't set the conversation id on inner flows.
• Don't set gen_ai.conversation.id or maple_ai.session.id on spans yourself; the processor does it.

Step 6: Content, errors, flush

• Content: Genkit always records genkit:input / genkit:output (full prompts, replies, tool args/results). There is no Genkit switch. If the user needs content kept out of Maple, delete genkit:input/genkit:output in the processor after mapping and skip the message/argument/result attributes; tell them transcripts will be empty.
• Never set OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT / OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT: a truncated genkit:input makes JSON.parse in the processor throw inside span.end().
• Tool failures: a tool that throws ends with status ERROR (Maple counts it failed) and aborts the whole generate / flow (Genkit doesn't feed the error back to the model). Don't change tool error behavior for tracing; tools that return { error } payloads show as successful, which is accurate to what the model saw.
• Streaming (ai.generateStream, flow.stream()): model spans end with the full output and usage; read the stream to the end before flushing.
• Flush:
  • Scripts/CLIs: await sdk.shutdown().catch((err) => console.error("telemetry flush failed", err)) in a finally. shutdown() rejects when an export failed; the .catch keeps a Maple outage from crashing the app.
  • Long-running servers (Cloud Run, plain Node): process.on("SIGTERM", () => sdk.shutdown().catch((err) => console.error("telemetry flush failed", err))); nothing per request.
  • Serverless handlers you control: await spanProcessor.forceFlush() after the flow returns (the flow span ends when the flow returns, so flushing inside the flow misses it).
  • Cloud Functions for Firebase onCallGenkit(flow): you can't hook after the flow; the instance may be throttled after the response, so the last batch can be delayed or lost. Tell the user; if it matters, wrap with onCall and call the flow then forceFlush() before returning. Untested.

Step 7: Verify

Run one conversation with 2+ turns under the same id, one tool call, plus a second conversation. If the app has no scriptable entry point (server, Developer UI only), write a small driver for this that calls the flow directly: one conversation id, 2+ turns, at least one tool call, flush before exit. Then check (Maple MCP list_agent_sessions / get_agent_session, or the Agent Sessions page, ~30 s after the run):

• One session per conversation (id = your conversation id), not trace:<id> sessions; the second conversation is separate.
• Framework shows Genkit.
• Turns = number of flow runs; each trace has invoke_agent (flow name) → generate → model (chat) and tool (execute_tool) spans.
• Transcript shows user messages, assistant replies, tool calls with arguments and results; turn labels are the user's messages.
• Every model span has input and output tokens (if the model plugin reports usage).
• A throwing tool is failed; successful tools are not.
• No attribute contains the provider API key or Bearer .
• Under genkit start, the Developer UI still shows traces (and nothing reaches Maple).
• The process exited cleanly and no turn is missing (flush ran).

Without Maple access: the run exits with no export errors on stderr (OTLPExporterError, Failed to export, 401 lines) AND a local check shows the spans. Silence alone proves nothing (no spans also looks silent). Local check: point OTEL_EXPORTER_OTLP_ENDPOINT at a small HTTP server and swap in @opentelemetry/exporter-trace-otlp-http with OTEL_EXPORTER_OTLP_PROTOCOL=http/json to read the spans as JSON; confirm gen_ai.operation.name on flow/model/tool spans and gen_ai.conversation.id equal across turns. With the Maple MCP: list_agent_sessions with search=<conversation id> returns one row.

Do not

• Do not add a second GenAI tracer that also exports to Maple (OpenLLMetry/OpenInference Google GenAI or OpenAI instrumentations for the same calls): model calls get recorded twice.