LangWatch is a testing and evaluation platform for AI agents, positioned to turn unpredictable agents into reliable production systems, with core capabilities including simulations, evals, observability, and governance.
Purpose
AI agent testing still largely relies on manual work, leading to frequent problems in production environments. An agent may achieve the same goal through hundreds of paths, and manual testing can cover only a few of them. LangWatch introduces "loop engineering," enabling teams to replace manual sampling with simulation-based continuous testing and evaluation, so that reliability gradually improves with every release. It is aimed at CTOs, engineers, AI architects, and product leaders who are bringing AI into critical production environments.
Main Capabilities
- Spec-driven agent building: automatically turn requirements into agent tests, accelerating development and establishing a self-improving agent loop.
- Reproduce issues from production: turn a production trace into a simulation to prove that a fix works.
- Simulate real users: simulated users converse with agents turn by turn in text and voice, applying pressure as in the real world; adversarial simulations can be combined to probe for jailbreaks, policy violations, and unsafe tool calls.
- Scenario writing: describe the behavior you want to test in natural language in Claude Code, and Scenario writes the tests; the same set of scenarios can be run during local development and in every pull request, with no separate configuration required.
- Evaluation: evaluators read the full trace like a human and unfold it step by step, giving conclusions with reasoning; supports scoring from a single output to an entire conversation, using LLMs, custom code, or full workflows; supports running evaluations while experimenting in a Jupyter notebook, and also in the UI for the whole team to use; supports online evaluation of production traffic, capturing arbitrary signals and turning them into insights; supports side-by-side comparison of two outputs to choose a better model, prompt, or version; also supports image and mixed-media evaluation.
- Tools and integrations: every tool call, skill, and MCP server is traced and can be mocked or fixtured for deterministic runs; supports white-box testing via API or black-box testing connected to internal implementations, compatible with all agent frameworks without rewriting.
- Observability: native support for the OpenTelemetry GenAI specification, with traces applicable to any frameworkCompatible with the OTel stack; provides Cmd+K quick jumps to any trace, filter, or view; tracks token, cost, and cache for each agent such as Claude Code, Codex, and opencode; supports saving custom views, natural language search, and intelligent filtering across millions of traces; provides multiple views including waterfall diagrams, flame graphs, topology diagrams, and sequence diagrams; automatically clusters each conversation by topic and subtopic; can build custom analytical charts based on any metric.
- AI assistance: Langy can turn a product manager's goals into a complete Scenario test plan and turn failures into pull requests, letting PMs own the spec while developers stay in flow.
How to use
LangWatch is deployed where the data resides, controls access permissions, and provides the authentication required for security review. Deployment options include: cloud (managed multi-tenant SaaS, covering EU/US/UK/APAC), self-hosted (Docker, Kubernetes/Helm, or your own VPC), and hybrid (data plane in your own infrastructure, control plane managed by LangWatch). Enterprise security controls include RBAC and REST API, cost center attribution, audit log forwarding to SIEM, custom retention policies, and ISO 27001 certification, EU data residency support, and monitoring by Vanta. The platform offers a 30-minute solutions engineer session to help onboard LangWatch end-to-end on the user's stack.
Use cases
Suitable for teams that need continuous testing and evaluation of AI agents, especially scenarios where agent logic is complex, there are many paths, and high reliability is required in production environments, such as simulating user conversations, adversarial security testing, online evaluation of production traffic, cross-framework observability, and organizations that need to meet enterprise procurement and compliance requirements.
Limitations
The materials do not provide specific billing methods or pricing information; do not provide free trial or version tiering details; do not provide a list of available SDK languages or code examples; and do not provide a specific list of supported frameworks, performance metrics, or implementation case data. The above content is not yet available.
