kubesense-traces

KubeSense Traces (APM)

Use for anything about requests: latency, p99/p95, error rate, HTTP method, status code, endpoint, service-to-service calls, downstream dependencies. (Log lines and panics → kubesense-logs.)

Requires the KubeSense MCP server; see kubesense-mcp. Tools resolve RBAC against the traces module.

Tools

Tool	Use when
`get-trace-or-log-fields` (`signal: "traces"`)	Always, first
`analyze-traces`	Latency percentiles, error rates, counts, trends
`search-traces`	Read individual spans
`get-distributed-trace`	You have a `trace_id` and need the full call chain

get-distributed-trace is the highest-leverage tool here. A single span tells you something failed; the trace tells you which hop failed and where the time went.

Field Names — Catalog Labels Only

Storage column names are rejected at every input slot.

Concept	Use this label	Storage column (rejected)
service identity	`service`	`app_service`
k8s workload	`workload`	—
pod	`instance`	`pod_name`
container	`container`	`container_name`
node	`node_name`	—
HTTP method	`method`	`subtype`
HTTP status code	`status_code`	`return_code`
resource / endpoint	`resource`	`clustered_resource`
span role	`role`	`kind`
protocol	`protocol`	`protocol_type`
span outcome	`status`	—
latency	`duration`	—
external call?	`request_type`	`is_external`
issue reason	`reason`	`issue_reason`
customer / tenant	`domain`	`customer_identifier`
trace / span id	`trace_id`, `span_id`	—
issue id	`issue_id`	—
namespace, cluster, region, app_version, env_type, source, server, client, server_namespace, client_namespace, operation_name, partner_cluster	same	—

Also parse but hidden from discovery: timestamp (→ start_timestamp), duration, primary_workload/primary_namespace (→ perspective_*), associate_workload/associate_namespace (→ partner_*).

`service` vs `workload`

Prefer service for application identity — it is the cross-platform identifier (Kubernetes, Docker, legacy). Use workload only when you specifically need Kubernetes topology.

[!NOTE] This is the opposite of the guidance for alert rules, where workload groups more reliably. See kubesense-alerts — the alert engine resolves service against a rollup column that is only populated for SDK-instrumented services.

[!WARNING] node is not valid on traces — it is node_name. (Logs are the reverse: node, not node_name.) Never carry a field name across signals.

There is no body field on traces. Text search over a message is logs-only.

Enum values are lowercase on traces

Field	Values
`status`	`error`, `ok`
`role`	`server`, `client`
`protocol`	`HTTP`, `gRPC`, `TCP`, `MongoDB`, `Redis`, `MySQL`, `PostgreSQL`
`method`	`GET`, `POST`, `PUT`, `DELETE`, `PATCH`
`source`	`eBPF`, `OTel`
`status_code`	`200`, `400`, `401`, `403`, `404`, `500`, `502`, `503`, `504`

status = error — lowercase, unlike logs' UPPERCASE type = ERROR.

status_code is a string field, so its values stay quoted strings even when they look numeric: status_code IN (500, 502) yields the strings "500", "502". Range comparisons therefore do not work — enumerate the codes, or use status_code LIKE "5%" for a class.

Read casing from the enum_values column, not example — the example generator returns a blank cell for method, protocol, and status_code.

Duration: ms in, ns out

[!IMPORTANT] Filtering duration takes MILLISECONDS. Aggregating duration returns NANOSECONDS. These are different units in the same field, and getting it wrong is silent.

where: "duration > 500"        →  spans slower than 500 milliseconds

The parser multiplies your value by 1e6 to reach the nanosecond storage column. So:

You want	Write
slower than 250ms	`duration > 250`
slower than 1s	`duration > 1000`
slower than 1.5s	`duration > 1.5e3` or `duration > 1500`

[!WARNING] The search-traces tool description shows "duration > 250000" as an example. That is wrong — it means 250,000 ms = 250 seconds and will match nothing. Ignore it.

Aggregation is the other direction — p95/avg/max over fields: [{"field": "duration"}] come back in nanoseconds:

Result	Means
`250000000`	250 ms
`1000000000`	1 s

Divide by 1e6 to report milliseconds. There is no duration_ms field at the MCP layer — it is rejected as an unknown field.

Trace attributes: group-by only

[!WARNING] @attr in a traces WHERE clause does not work. It is rejected with unknown field "<key>" for signal=traces, because the parser clears the attribute discriminator during the ARRAY_MAP remap and the field then fails column validation. The tool descriptions advertise @db.system = postgresql as working; it does not.

Trace attributes do work everywhere the field is passed structurally — set the flag instead of a prefix:

"group_by_fields": [ { "field": "db.system", "is_attribute": true } ]

Valid in group_by_fields, fields, required_fields, and sort_by. To filter by an attribute value, group by it and read the series labels, or filter on an equivalent top-level column.

(Log attributes are unaffected — @user.id = abc works fine on logs.)

analyze-traces

{
  "from_time": "2026-07-30T10:00:00Z",
  "to_time": "2026-07-30T11:00:00Z",
  "query_type": "range",
  "group_by_fields": [ { "field": "service", "is_attribute": false } ],
  "fields": [ { "field": "duration", "is_attribute": false } ],
  "value_operation": "p95"
}

value_operation: row_count, unique_count, avg, sum, min, max, p50, p90, p95, p99. count is not valid — it is row_count, despite what the schema's summary line suggests.

Question	Shape
p95 latency by service, over time	`range`, group `service`, `fields: [duration]`, `p95`
top 5 services by errors, now	`instant`, `where: "status = error"`, group `service`, `row_count`, `sort_direction: DESC`, `limit: 5`
errors by status code	`instant`, `where: "status = error"`, group `status_code`, `row_count`
slowest endpoints	`instant`, group `resource`, `fields: [duration]`, `p99`, `sort_direction: DESC`
server-side only	add `where: "role = server"` — otherwise client spans double-count

Error rate as a percentage needs two queries plus a formula → analyze-telemetry. See kubesense-mcp.

Output is columnar JSON: for range, one shared timestamps array plus a positionally aligned values array per series; for instant, one scalar value per series.

search-traces

Same input shape as search-logs. Output is TSV with a # total_count=<N> is_free_search=<bool> header.

Pagination — and its hard limit

Page by narrowing to_time to the minimum start_timestamp returned (not timestamp, which is the span's end time; start_timestamp is the sort and filter key). The window is half-open, so the cursor row is not repeated.

[!WARNING] Exhaustive paging is not achievable for traces. Rows are ordered toStartOfHour(start_timestamp) DESC, observation_point DESC, cluster DESC, start_timestamp DESC. Both observation_point and cluster outrank the timestamp, so pinning clusters alone is not enough — spans at other observation points in the same hour get skipped. And observation_point is not in the trace field catalog, so the where clause cannot pin it (the tool description suggests "observation_point = s-app"; that is rejected as an unknown field).

Practical consequences: keep the window short enough that total_count fits in one page, or use analyze-traces. Treat any paged trace scan as a sample.

Touching duration, trace_id, span_id, or any attribute sets is_free_search and routes to the raw table — slower, no rollups.

get-distributed-trace

The span tree for one request. Start here when investigating a failing or slow request.

{
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "from_time": "2026-07-30T10:20:00Z",
  "to_time": "2026-07-30T10:40:00Z",
  "max_spans": 200
}

trace_id, from_time, to_time are all required. The window must contain the trace's start_timestamp — traces are time-partitioned, so a window that misses it finds nothing. Use ~10 minutes either side. max_spans defaults to 200, max 500.

[!NOTE] Key on trace_id, not a span's kuid. One trace has many spans; they all share the trace_id.

Reading the output

Block 1 — one row per span in preorder (a span is immediately followed by its children), with a tree column drawing the call chain. Key columns:

Column	Read it for
`tree`	who called whom
`status` / `return_code`	which span failed
`duration_us`	total span time in microseconds, including everything it called
`self_time_us`	time in this service, excluding children — the bottleneck ranking
`start_time` / `end_time`	RFC3339 ms — paste straight into `search-logs`
`workload` / `namespace` / `cluster`	scope for follow-up log/metric queries
`resource_name` / `protocol_type` / `subtype`	what was called

Block 2 — ## services, one row per (cluster, workload), ordered worst-first by errors then self time. Read this before the span rows to pick your suspect. It always covers every span, even when block 1 was truncated.

Two diagnostic rules

A trace with no error may still be the problem. The top row of ## services is the hop that burned the time.
Large duration_us + small self_time_us = the service was waiting, not working. Investigate what it called (its children), not the service itself. Conversely, if a span errored but every child succeeded, the fault is in that service.

namespace may be blank when spans are attributed to a collector agent rather than the workload's pod; take it from a span row that has one, or from get-workload-detail. Rarely it lists several joined by |, meaning that cluster runs the same workload name in more than one namespace.

Investigation Flow

1. get-distributed-trace(trace_id, ±10min)   → the call chain, errors marked
2. read "## services"                        → who owns the errors / the time
3. search-logs                               → scoped to that workload + the failing
                                               span's start_time/end_time (widen a few s)
4. get-recent-changes / get-infra-issues     → why: deploy, OOM, probe failure
5. analyze-metrics                           → confirm resource pressure

Steps 4–5 use kubesense-infra and kubesense-metrics.

Rules

get-trace-or-log-fields with signal: "traces" first, same window as the query.
Catalog labels only: service, instance, method, status_code, resource, role, protocol. Never app_service, pod_name, subtype, return_code, clustered_resource, kind, protocol_type.
node_name on traces, node on logs. Never carry a field across signals.
Prefer service over workload for app identity — but the alert engine prefers workload.
duration in WHERE is milliseconds; aggregated duration is nanoseconds. Divide results by 1e6 to report ms. duration_ms does not exist here.
@attr filters do not work on traces. Use is_attribute: true in group_by_fields/fields instead.
Enum values are lowercase (status = error, role = server); status_code is a string, so enumerate codes or use LIKE "5%" rather than a range.
value_operation is row_count, not count.
Add role = server when counting requests, or client spans double-count.
get-distributed-trace needs a window containing the trace's start; widen it first if nothing is found.
Don't try to page traces exhaustively — observation_point can't be pinned. Shorten the window or aggregate.