TruLens is an open-source tool for evaluating and tracing LLM applications and AI agents, with native support for OpenTelemetry. Its core goal is to move developers from judging by feel to making decisions based on metrics, identifying where agents fail and where costs can be reduced without sacrificing quality.
Main capabilities
- Tracing and instrumentation: Instrument AI agents using OpenTelemetry, recording every span, including latency, inputs, outputs, tokens, and cost, recorded step by step so that wrong answers have traceable causes.
- Evaluation and scoring: Score each step using benchmarked LLM judges; the judges explain the basis for their scores, and low scores can directly point to the specific step that went wrong. The official description says its judges perform excellently when compared against human-annotated scores, and can be adapted to specific domains using private data, scoring criteria, examples, and score range adjustments.
- Version comparison: Supports comparing different versions, helping find the frontier of quality and cost and decide which version should be released.
- Evaluation dimensions: Including whether the right tools were selected and the plan was followed, execution efficiency, whether answers are supported by retrieved content, context relevance, answer relevance, whether the MCP server returns usable results, MCP span tracing, whether substantive content is retained and noise removed, comprehensiveness, logical consistency, retrieval and RAG, ranking (NDCG, MRR), precision/recall, etc.
- Flexible integration: Supports any Python function, any LLM-as-judge prompt, any span attribute; can trace already-built applications, use a model of your choice for judging, and write to a self-run database, with no rewriting or lock-in.
How to use
- You can instrument any application in pure Python, or automatically instrument common frameworks.
- Supports running in real time or running based on a dataset: record one trace to start evaluating; run a dataset to replay data. Both methods use the same metrics, and evaluation spans fall into the trace, scoring, and leaderboard views.
- Provides quick starts for LangChain, LangGraph, LlamaIndex, and custom applications.
Applicable scenarios
- Development teams that need to locate the causes of AI agent failures, optimize the balance between cost and quality, and compare the effects of different versions.
- Applications using complex agent architectures such as RAG, retrieval, tool calling, and MCP servers.
Limitations and notes
- The materials do not provide specific fees, prices, platform support scope, or detailed installation commands; this information is not currently provided.
- The project was originally created by TruEra and is now actively maintained by Snowflake in the open-source community.
