SuperAPI AI Directory

judge-reliability-checker

General & other

View repository

Updated 10-16 06:44

About this skill

Judge Reliability Checker

Evaluate the evaluator.

Workflow

  1. State what the judge is deciding.
  2. Test prompt sensitivity and ordering effects.
  3. Check agreement with trusted human labels if available.
  4. Look for verbosity, position, or style bias.
  5. Recommend whether the judge is usable, limited, or unsafe.

Rules

  • Do not treat one judge prompt as ground truth.
  • Bias discovered in evaluation is a result, not a nuisance.