Frozen benchmarks, hidden tests, Pareto front, and evaluation standards.
When to Use
- Initialize or query the benchmark database
- Compare skill scores against the frozen baseline
- Check the Pareto front (reject if any dimension regresses >5%)
- Consult the quality tier standards (POWERFUL/SOLID/GENERIC/WEAK)
- Add new frozen test cases to the benchmark library
- View the historical best score of a skill across all dimensions
- Provide Pareto baseline data for the RegressionGate of improvement-gate
- List all registered benchmark entries in a batch evaluation scenario
When NOT to Use
- Score candidates → use
improvement-discriminator - Automatic improvement → use
improvement-learner - Full pipeline → use
improvement-orchestrator - Execute changes → use
improvement-executor - Gate decision → use
improvement-gate(it consumes data from benchmark-store)
Quality Tiers
| Tier | Score | Ship? |
|---|---|---|
| POWERFUL ⭐ | ≥ 85% | Marketplace ready |
| SOLID | 70–84% | GitHub |
| GENERIC | 55–69% | Needs iteration |
| WEAK | < 55% | Reject or rewrite |
Tiering is based on a weighted composite score across all dimensions. Each dimension has an independent weight in evaluation-standards.md. accuracy and coverage have the highest weights (0.2 each), security has a weight of 0.15, and the remaining dimensions each have 0.1-0.15. Tiering is used by improvement-orchestrator to decide whether to continue iterating: POWERFUL means stop, WEAK requires a retry.
Why Pareto Front Instead of Weighted Average
Tradeoff: Using a weighted average score to judge regression seems simple (comparing a single number), but it hides trade-offs between dimensions. For example, if accuracy drops from 0.9 to 0.6 while coverage rises from 0.5 to 0.8, the weighted average may stay flat or even rise, but the large regression in accuracy is masked. Because the Pareto front checks dimension by dimension, any single dimension regressing beyond the 5% threshold triggers rejection, ensuring that improvements are true Pareto improvements (no dimension gets worse) rather than pseudo-improvements at the cost of sacrificing some dimension.
The cost of this design is that improvements are harder to accept (pass rate about 40-50%), but it avoids the drift problem of "getting more biased the more you change." In autoloop scenarios, Pareto protection is the key mechanism preventing the LLM from gradually losing existing advantages over multiple rounds of iteration.
Pareto Front
ParetoFront.check_regression(new_scores) → {"regressed": bool, "regressions": [...]}
# 5% tolerance — minor fluctuations allowed
The complete flow of the Pareto regression check: load the historical best scores → compare dimension by dimension → mark dimensions exceeding the 5% threshold as regression → any single regression means the overall regressed=True.
# Pareto regression check — full example
from lib.pareto import ParetoFront
pf = ParetoFront("state/pareto.json")
new_scores = {"accuracy": 0.82, "coverage": 0.90, "reliability": 1.0,
"efficiency": 0.95, "security": 1.0, "trigger_quality": 0.75}
result = pf.check_regression(new_scores)
# result = {"regressed": True,
# "regressions": [{"dim": "accuracy", "before": 0.90, "after": 0.82, "delta": -0.08}]}
# accuracy dropped 8% (> 5% threshold) → regression detected → candidate rejected
CLI
# List benchmarks
python3 scripts/benchmark_db.py --action list --db-path benchmarks.db
# Compare skill against baselines
python3 scripts/benchmark_db.py --action compare --skill-path /path/to/skill --category general --db-path benchmarks.db
# Add a benchmark
python3 scripts/benchmark_db.py --action add --category general --test-name "test1" --db-path benchmarks.db
Output Artifacts
| Request | Deliverable |
|---|---|
| Init | SQLite database with schema |
| Compare | JSON comparison with per-dimension delta |
| Pareto check | JSON with regressed flag and details |
| List | All registered benchmarks with metadata |
| Add | Confirmation of new benchmark insertion |
The Compare output includes a before/after/delta triple for each dimension, as well as the overall tier_before/tier_after tier changes. When upgrading from SOLID to POWERFUL or downgrading from GENERIC to WEAK, it additionally marks tier_changed: true.
Related Skills
- improvement-learner: Imports ParetoFront for self-improvement loop
- improvement-gate: RegressionGate uses Pareto data to block regressions
- improvement-discriminator: References evaluation standards for scoring context
- improvement-orchestrator: Full pipeline queries benchmark-store at gate stage
- autoloop-controller: Uses historical benchmark data to detect convergence plateau
Data Files
data/evaluation-standards.md— Quality tiers, dimensions, weights (v2.0.0)data/fixtures/— Frozen test fixturesstate/pareto.json— Per-skill Pareto front historical best scoresbenchmarks.db— SQLite database storing all benchmark entries and results