sabot
AI agentsSabot is an open-source benchmark project for evaluating the effectiveness of self-check mechanisms in multi-agent pipelines. It injects controlled faults into reference pipelines built with frameworks such as LangGraph, CrewAI, and AutoGen/Magentic-One, measuring whether the pipelines' own reviewer and guardrail stages can detect, respond to, or miss these faults. Test results show a median detection rate of 16.7%, and by modifying the reviewer prompt to require it to report all inconsistencies, the detection rate can be increased to 55.0%. The project provides frozen measurement specifications, runnable test frameworks, and public raw run traces, and is licensed under Apache-2.0.