sabot

Sabot is an open-source project for studying and measuring the effectiveness of self-check mechanisms in multi-agent pipelines. It is not a production tool, but a combination of a standard, a test framework, and a public scoreboard, used to answer a specific question: can the agent pipeline's own checks catch injected faults?

Purpose and Goals Sabot aims to evaluate the ability of built-in reviewers, critic stages, and guardrail callbacks in multi-agent pipelines built on frameworks such as LangGraph, CrewAI, and AutoGen/Magentic-One to detect, respond to, and recover from controlled faults. The project injects faults into running pipelines and scores whether the pipeline's own mechanisms flag, handle, or miss these faults.

Main Capabilities

• Fault Injection and Scoring: Inject six types of fault operators into reference pipelines (such as tools returning incorrect values, handoffs changing, documents becoming stale, review steps being skipped, and models silently degrading), and give a three-level verdict for each fault: DETECTED, REACTED, RECOVERED.
• Hard Hierarchical Detection Rate: Only counts detection behavior that is surfaced by the framework itself and deterministically adjudicated, forming the core Sabot score.
• Offline Reproduction: No API keys, model calls, or virtual environments are required. The repository contains the raw traces of all 2,025 runs, and the scores can be recomputed offline using Python 3.9+ standard library scripts.
• Publicly Pre-registered Specifications: Specifications, anchors, and predictions are published before scoring runs, and negative results are released with equal prominence.
• Uncertainty Quantification: Provides interval estimates for all metrics via repeated clustered bootstrap, and explicitly labels them as a sensitivity analysis of five repetitions rather than random seed reproduction.

Usage

• Offline reproduction: under harness/scripts/ run python scripts/score_by_operator.py --check to verify the table categorized by operator; run python scripts/score_strict.py to regenerate the strict evidence lower-bound table.
• Run the test framework: requires Python 3.11+ and a pinned framework virtual environment (see harness/pyproject.toml and harness/scripts/setup_venvs.sh).
• All original trajectories and scoring scripts are released in the repository, allowing anyone to independently verify the results.

Applicable scenarios

• Study the effectiveness of self-checking and guardrail mechanisms in multi-agent systems.
• Compare the performance patterns of different agent frameworks in fault detection.
• Serve as a reference for understanding the design limitations of reviewers when developing or evaluating agent pipelines.
• Used to verify or reproduce metrics such as detection rate, response rate, and recovery rate released by the project.

Limitations and notes

• The results are based on a specific reviewer design and a single pipeline model (gpt-5.6-terra) and do not represent framework rankings or universal laws.
• The findings mainly focus on the fact that textual faults can be noticed, while process faults mostly go undetected; noticing is not the same as taking action.
• Silent model degradation (O4) did not show a detection signal, and its flagging rate was comparable to the false positive rate of the clean baseline.
• Hard-tier detection has a deterministic design, but the soft tier relies on an LLM judge (kappa 0.767).
• The project explicitly states that its published results are not changed according to expectations, and negative results are also made public.
• More details about the project's pricing standards, commercial support, and specific installation steps are not currently available in the provided materials.