jaeger

Jaeger — Local Trace Inspection

Gram runs a local OpenTelemetry Collector that receives all OTLP signals from gram-server and gram-worker, routing traces to Jaeger and metrics to Prometheus. Everything starts automatically with mise run infra:start.

Architecture

App → OTLP :$OTLP_GRPC_PORT → OTel Collector → traces → Jaeger
                                               → metrics → Prometheus (scrapes collector)
                                               → spanmetrics connector → Prometheus (RED metrics from traces)

OTel Collector receives all OTLP (traces + metrics) on $OTLP_GRPC_PORT
Jaeger receives traces from the collector (not directly from the app)
Prometheus scrapes the collector's metrics exporter and stores both app metrics and span-derived RED metrics

Discovering Ports

Jaeger ports are configured via environment variables in mise.toml. Always resolve ports from env vars — never hardcode them, as they may differ across worktrees or local overrides.

# Jaeger UI/API port
echo $JAEGER_WEB_PORT

# OTLP gRPC receiver port
echo $OTLP_GRPC_PORT

Web UI: http://localhost:$JAEGER_WEB_PORT
OTLP gRPC receiver: localhost:$OTLP_GRPC_PORT
REST API: http://localhost:$JAEGER_WEB_PORT/api/...

Jaeger REST API

Use these endpoints to programmatically query traces after running seed data or hitting endpoints. Replace $JAEGER_WEB_PORT with the resolved value.

List services

GET http://localhost:$JAEGER_WEB_PORT/api/services

Returns all instrumented services (e.g., gram-server, gram-worker).

Search traces

GET http://localhost:$JAEGER_WEB_PORT/api/traces?service=gram-server&limit=20
GET http://localhost:$JAEGER_WEB_PORT/api/traces?service=gram-server&operation=POST /v1/mcp/{mcpSlug}&limit=10
GET http://localhost:$JAEGER_WEB_PORT/api/traces?service=gram-server&tags={"http.status_code":"500"}&limit=10

Query parameters:

service (required) — service name
operation — filter by operation/endpoint
tags — JSON object of tag key-value filters
start / end — microsecond Unix timestamps
limit — max traces returned (default 20)
minDuration / maxDuration — e.g., 1ms, 500ms, 2s
lookback — e.g., 1h, 2h, 1d

Get a specific trace

GET http://localhost:$JAEGER_WEB_PORT/api/traces/{traceID}

Returns all spans for a trace, including cross-service spans between gram-server and gram-worker.

List operations for a service

GET http://localhost:$JAEGER_WEB_PORT/api/services/{service}/operations

Returns all known operation names (HTTP routes, gRPC methods, Temporal activities).

Development Workflow

Start infra — mise run infra:start (Jaeger starts automatically)
Start server — mise start:server --dev-single-process
Run seed data or hit endpoints — exercise the code path you're working on
Query Jaeger — use the API or UI to inspect the resulting traces
Look for: slow spans, error spans (error=true tag), missing instrumentation, N+1 query patterns

Trace Structure

The Gram server uses OpenTelemetry with these conventions:

HTTP spans: created by otelhttp middleware, operation = HTTP <method> <route>
Database spans: created by pgx tracing, operation = SQL query
Temporal spans: activity and workflow execution spans
Custom spans: created via tracer.Start(ctx, "operation-name") in service code

Key attributes on spans:

http.method, http.route, http.status_code — HTTP request details
db.statement — SQL query text
service.name — gram-server or gram-worker
error — true if the span recorded an error

Prometheus — Local Metrics

Prometheus is available at http://localhost:$PROMETHEUS_PORT. It stores:

App metrics — custom counters, histograms, gauges exported via OTLP from server/worker
Span metrics — RED metrics (rate, errors, duration) derived from traces by the OTel Collector's spanmetrics connector

# Prometheus UI
echo $PROMETHEUS_PORT

# Example PromQL queries
# Request rate by service:    calls_total{service_name="gram-server"}
# Error rate:                 calls_total{status_code="STATUS_CODE_ERROR"}
# P99 latency:                histogram_quantile(0.99, sum(rate(duration_milliseconds_bucket[5m])) by (le, service_name))

Environment Variables

Configured in mise.toml:

Variable	Default	Purpose
`OTLP_GRPC_HOST`	`localhost`	OTLP receiver host (OTel Collector)
`OTLP_GRPC_PORT`	`4317`	OTLP receiver port (OTel Collector)
`OTEL_EXPORTER_OTLP_ENDPOINT`	`http://localhost:4317`	Full endpoint (auto-constructed)
`JAEGER_WEB_PORT`	`16686`	Jaeger UI/API port (traces)
`PROMETHEUS_PORT`	`9099`	Prometheus UI/API port (metrics)
`GRAM_ENABLE_OTEL_TRACES`	`1`	Enable/disable trace export
`GRAM_ENABLE_OTEL_METRICS`	`1`	Enable/disable metrics export