evidently

Evidently is an open-source Python library for evaluating, testing, and monitoring machine learning (ML) and large language model (LLM) systems, covering the complete workflow from experimentation to production.

Purpose Evidently is designed to provide observability for AI-driven systems or data pipelines, supporting tabular data and text data, and is suitable for predictive tasks and generative tasks (such as classification and RAG).

Main capabilities

• Provides more than 100 built-in metrics, covering data drift detection, LLM judging, and more.
• Supports a Python interface for custom metrics.
• Supports both offline evaluation and real-time monitoring.
• Uses an open architecture, making it easy to export data and integrate with existing tools.
• Modular design: you can start with one-off evaluations or build a complete monitoring service.

How to use

1. Reports and test suites: Reports calculate and summarize various data, ML, and LLM quality evaluations. You can use presets and built-in metrics, or customize them. Suitable for experiments, exploratory analysis, and debugging. Reports can be viewed interactively in Python, or exported as JSON, Python dictionaries, HTML, or viewed in the monitoring UI. By adding pass/fail conditions, any report can be turned into a test suite, suitable for regression testing, CI/CD checks, or data validation. Test conditions support simple syntax such as gt (greater than) and lt (less than), and test conditions can be automatically generated from a reference dataset with zero setup.
2. Monitoring dashboard: The monitoring UI service helps visualize metrics and test results over time. You can choose the self-hosted open-source version (an online demo is available), or sign up for Evidently Cloud (recommended), which offers a generous free tier as well as additional features such as dataset and user management, alerts, and no-code evaluation.

Installation Install from PyPI: pip install evidently Install using the Conda installer: conda install -c conda-forge evidently

Quick start LLM evaluation example: import the necessary components, create a toy dataset containing questions and answers, create an Evidently Dataset object and add descriptors (row-level evaluators), for example checking the sentiment, length, and whether each response contains words indicating denial. You can view the dataframe with added scores and generate a summary report to see the score distribution. You can also choose other evaluators, including LLM-as-a-judge, and configure pass/fail conditions.

Data and ML evaluation example: importReport, evaluation presets, and a toy tabular dataset, run the data drift evaluation preset, and test for shifts in column distributions. Use the first 60 rows of the dataframe as the "current" data and the rest as reference data. You can get the output in a Jupyter notebook, or save it as an HTML file (which needs to be opened from the target folder), or output it as JSON or a Python dictionary. You can also choose other presets, create reports from a single metric, and configure pass/fail conditions.

Monitoring dashboard Launch a demo project in the locally hosted Evidently UI. Signing up for Evidently Cloud gives you immediate access to a hosted version with additional features. If uv is already installed, you can run the Evidently UI with a single command; otherwise, create a virtual environment, install Evidently, and then run the UI with the demo project, and visit localhost:8000 to use it.

Use cases

• Data science and machine learning teams performing model evaluation and data drift detection.
• Output quality evaluation and RAG evaluation in LLM application development.
• Model performance monitoring and alerting in production environments.
• Automated testing and data validation in CI/CD workflows.

Limitations The materials do not clearly specify Evidently's specific limitations, system requirements, performance limits, or complete functional boundaries.