improvement-discriminator

Improvement Discriminator

Multi-signal scoring engine: heuristic rules + evaluator rubrics + LLM-as-Judge + multi-reviewer blind panel.

When to Use

Score and rank improvement candidates—produce a ranked list for the executor to execute by priority
Run multi-reviewer blind review (CONSENSUS/VERIFIED/DISPUTED epistemic labels) to reduce single-reviewer bias
Use LLM-as-Judge to evaluate 4 semantic dimensions (clarity, specificity, consistency, safety)
Combine --panel + --llm-judge for the most comprehensive evaluation (the two are not mutually exclusive)
Debug why all candidates are marked as hold—usually risk_penalty is too high or source_refs are missing
Automatically invoked in stage 2 of the orchestrator pipeline
When you need explainable scoring details (each dimension scored independently, with judge_notes)
When you need to compare candidate quality trends across multiple rounds of improvement

When NOT to Use

Evaluate skill directory structure → use improvement-learner(learner does 6-dimension structural analysis, discriminator does semantic scoring)
keep/revert/reject decisions → use improvement-gate(discriminator only scores and does not make decisions)
Execute file changes → use improvement-executor(discriminator does not modify any files)
Generate improvement candidates → use improvement-generator
Calling it when the candidate count is 0 will raise an error; first confirm that the generator produced candidates
Do not use discriminator for regression detection—regression detection is handled by the gate's RegressionGate layer
Do not treat discriminator's accept as the final decision—it is only a scoring recommendation; the final keep/reject is decided by the gate
Do not call it without candidates.json—the input must be in the standard format produced by the generator

Scoring Modes

The 4 modes can be used independently or in combination, progressively increasing evaluation depth from pure heuristics to full signal fusion. The default mode (Heuristic only) is zero-cost and deterministic, suitable for fast filtering. After adding evaluator evidence, task_suite execution results are used as the scoring basis, which is closer to actual performance. After adding LLM Judge, semantic understanding is introduced, but it consumes tokens and the results have randomness. Panel mode introduces multi-perspective blind review, capturing biases that a single reviewer cannot find.

Mode	Flag	Scoring
Heuristic only	(default)	category bonus + source refs + risk penalty
+ Evaluator	`--use-evaluator-evidence`	Heuristic 70% + evaluator 30%
+ LLM Judge	`--llm-judge {claude,openai,mock}`	Heuristic 60% + LLM 40%
+ Panel	`--panel`	2+ reviewers independently, cognitive label decides
All combined	`--panel --llm-judge mock --use-evaluator-evidence`	Full

Why Panel Review Matters

Tradeoff: single reviewer speed vs. multi-reviewer accuracy.

The reason for introducing panel blind review rather than relying on a single scorer is:

Single reviewer bias is real — a "structural" reviewer tends to give high scores (preferring structurally complete candidates), while a "conservative" reviewer tends to give low scores (preferring minimal changes). A single reviewer will systematically favor a certain type of candidate.
CONSENSUS/VERIFIED/DISPUTED three-state labelscapture the degree of disagreement among reviewers. DISPUTED means the candidate's quality is controversial, and the gate layer will make a more cautious decision based on this.
LLM-as-Judge compensates for heuristic blind spots — heuristic rules cannot determine "whether this rewrite is really clearer", and the LLM judge's clarity/specificity dimensions fill this gap.

Question: Why not just use an LLM judge to replace all heuristic rules? Because LLM judge has token costs (about 500-1000 tokens per candidate) and has randomness. Heuristic rules are deterministic and zero-cost, suitable as the first layer of filtering. When used in combination, heuristics account for 60% of the weight and the LLM for 40%, ensuring both stability and semantic understanding.

When all panel results are hold, it is usually one of the following reasons:

The candidate lacks source_refs(citation source), causing source_ref_bonus = 0
risk_level is marked as high, causing risk_penalty to be too large
The LLM judge gave a low score on the safety dimension (the candidate may have introduced unsafe patterns)

CLI

score.py is the core entry point, receiving candidates.json and outputting scored.json. All modes share the same entry point, controlling scoring depth through flag combinations. --llm-judge Supports 3 providers: claude (most accurate), openai, mock (for testing, zero cost). --panel Automatically creates two independent reviewers, structural and conservative. The output scored.json can be passed directly to executor or gate for consumption. Use --verbose to view each reviewer's detailed scoring process and judge_notes.

# Basic scoring (heuristic only, fastest)
python3 scripts/score.py --input candidates.json --output scored.json

# Full pipeline: panel + LLM judge
python3 scripts/score.py \
  --input candidates.json --panel --llm-judge mock --output scored.json

Panel-only mode (no LLM, lower cost):

# Panel blind review without LLM judge — heuristic scoring only
python3 scripts/score.py \
  --input candidates.json \
  --panel \
  --output scored.json

LLM-judge-only mode (no panel, single reviewer):

# Single reviewer + LLM semantic evaluation
python3 scripts/score.py \
  --input candidates.json \
  --llm-judge claude \
  --use-evaluator-evidence \
  --output scored.json

Output Artifacts

Request	Deliverable
Score	JSON: per-candidate scores, blockers, recommendations, judge_notes
Panel	JSON: panel_reviews[], cognitive_label (CONSENSUS/VERIFIED/DISPUTED), aggregated_score
LLM judge	JSON: llm_verdict with score, decision (accept/conditional/reject), 4 dimensions, confidence
Combined	All above fields merged into a single scored candidate object

In the output, cognitive_label meaning:

CONSENSUS — All reviewers agree in scoring direction (all accept or all reject), high confidence
VERIFIED — The majority of reviewers agree, with a minority having reservations, medium confidence
DISPUTED — There is fundamental disagreement among reviewers (one accepts, one rejects), requiring extra careful handling at the gate layer

decision The field's three-state semantics: accept passes directly, conditional requires satisfying additional conditions (recorded in judge_notes), reject rejects directly.

Related Skills

improvement-generator: Produces the candidates that this skill scores — the discriminator's input comes from the generator
improvement-gate: Consumes scored candidates for keep/revert/reject — the gate relies on cognitive_label for ReviewGate decisions
improvement-learner: Structural evaluation (6-dim); the discriminator focuses on semantic — the two are complementary, the learner looks at structure, the discriminator looks at content
benchmark-store: Frozen benchmarks for regression checking — provides historical baseline data
improvement-executor: Applies top-ranked candidates — the executor executes in order according to the discriminator's ranking
improvement-evaluator: Task-based evaluation — the evaluator's pass_rate can serve as --use-evaluator-evidence data source
improvement-orchestrator: Calls discriminator as stage 2 — in the full pipeline, the discriminator comes after the generator and before the evaluator

Data flow in the Pipeline: generator → discriminator → evaluator → executor → gate