AlpacaEval is an automatic evaluator for instruction-following language models, developed and maintained by Tatsu-lab. Its official page is https://tatsu-lab.github.io/alpaca_eval/。
Purpose: AlpacaEval aims to provide a fast, low-cost, and reliable automatic evaluation method based on large language models, for comparing the ability of different models to follow general user instructions. It is based on the AlpacaFarm evaluation set, which tests a model's ability to follow general user instructions.
Main capabilities: AlpacaEval evaluates models by comparing model-generated responses with reference responses. For AlpacaEval, the reference responses come from Davinci003; for AlpacaEval 2.0, the reference responses come from GPT-4 Preview. The comparison is performed by the provided GPT-4-based automatic annotator, ultimately yielding a win rate. The official documentation states that AlpacaEval has a high agreement rate with real human annotations, and its leaderboard rankings are highly correlated with leaderboard rankings based on human annotations.
Usage: Community members can contribute new models to the leaderboard. The specific steps include: running the model on the evaluation set, automatically annotating the outputs, and then submitting a PR containing the model configuration and leaderboard results. The official team also provides Discord for community support and discussion. In addition, the community can also contribute new evaluators or new evaluation sets. For new evaluators, the official team has released real human annotations and comparison metrics; for new evaluation sets, the official team has released rough guidelines. The official team especially encourages contributions of harder instruction distributions and safety tests for large language models.
Applicable scenarios: AlpacaEval is suitable for scenarios that require fast, low-cost comparison of the capabilities of different instruction-following language models, such as iterative evaluation during model development and leaderboard ranking comparisons.
Limitations: AlpacaEval is not a comprehensive or gold-standard evaluation of model capabilities. According to the AlpacaFarm paper, the automatic annotator's win rate is correlated with response length. Although human annotations also exhibit this bias, it is unclear whether more verbose responses increase utility in downstream tasks. In addition, although the AlpacaFarm evaluation set is diverse, it mainly consists of simple instructions. The official team encourages the community to contribute new, more complex evaluation sets, such as evaluation sets for tool use. Finally, AlpacaEval does not evaluate the safety of any model.
