SuperAPI AI Directory

llm-benchmarks

General & other

View repository

Updated 11-17 16:26

About this skill

LLM Benchmarks — A Critical Guide

Benchmarks are the industry's shared scoreboard — and widely misread. This skill covers what the major benchmark families actually measure, why scores mislead, and how to use benchmarks without fooling yourself.

Overview

Benchmark families: knowledge (multi-choice QA across subjects), reasoning (math, logic, code — often with verifiable answers), instruction-following (did it do what was asked, how asked), long-context (retrieval and reasoning over long inputs), multilingual (quality beyond English), and agentic (multi-step tool use). Each measures something real and something narrow. The critical skills: knowing which family matches your use case, discounting for contamination (benchmark questions in training data), and never mistaking a leaderboard for a buying decision.

When to use

  • Choosing between models for a task: which benchmarks are actually relevant.
  • Reading vendor benchmark claims critically.
  • Designing your own evaluations informed by benchmark methodology.
  • Understanding what a new benchmark release does and doesn't prove.

Core concepts

  • Benchmark families: knowledge QA, reasoning/math/code, instruction following, long-context, multilingual, agentic/tool-use, safety/refusal. Map your task to families; ignore the rest.
  • What's measured vs. what's claimed: a multi-choice science score measures test-taking on science questions, not scientific judgment. Interpret literally.
  • Contamination: benchmark items leaking into training data inflates scores. Suspect unusually high scores on popular benchmarks; prefer fresh or private evals for decisions.
  • Saturation: benchmarks get maxed out and stop discriminating. A 95% vs 97% gap on a saturated benchmark means little; look at harder or newer tests.
  • Variance: small benchmarks have wide confidence intervals; single-run scores wobble. Ask for error bars or multiple runs before concluding.
  • Your evals beat public ones: public benchmarks measure general capability; your 50 task-specific cases measure what you actually need. Always build your own.

Practical workflow

  1. Define your task; identify the 1–2 benchmark families most related — treat the rest as background.
  2. Read benchmark claims critically: which version, what prompting, how many shots, any contamination notes?
  3. Shortlist models on relevant benchmarks, then evaluate finalists on your own task-specific set.
  4. For high-stakes decisions, prefer private/fresh evals over public leaderboard positions.
  5. Track benchmarks over time for the models you use — but re-validate on your tasks when models update.
  6. Document which benchmarks informed each model choice, so the decision is auditable.
Benchmark reading checklist:
[ ] Which family? Relevant to my task?
[ ] Version, shots, prompting disclosed?
[ ] Contamination addressed?
[ ] Saturated? (near-ceiling scores = weak signal)
[ ] Variance reported?
[ ] Validated against MY eval set before deciding?

Common pitfalls

  • Leaderboard shopping: picking the #1 model without checking relevance to your task. Rank ≠ fit.
  • Ignoring contamination: treating inflated scores as real capability. Discount popular benchmarks.
  • Single-number decisions: one benchmark score choosing a model. Triangulate across families + your evals.
  • Saturated benchmark worship: celebrating 0.5% gains at the ceiling. Noise, not signal.
  • No task-specific eval: shipping on public scores alone. Your data, your distribution, your eval.
  • Benchmark as safety proof: capability benchmarks don't measure safety. Separate safety evals are their own discipline.