InstructEval is a codebase for evaluating instruction-tuned large language models, with the goal of providing simple and convenient benchmarking across multiple tasks and models.
Purpose The project is used to quantitatively evaluate instruction-tuned models, such as Alpaca and Flan-T5, on held-out tasks. Instruction-tuned models are considered a direction for approaching the performance of large language models such as ChatGPT at lower cost, but performance across different models is difficult to compare qualitatively, so this project uses academic benchmarks to measure the generalization ability of models on a large number of unseen and challenging tasks.
Main capabilities Supports benchmarking multiple models, covering most models in HuggingFace Transformers:
- AutoModelForCausalLM: such as GPT-2, GPT-J, OPT-IML, BLOOMZ
- AutoModelForSeq2SeqLM: such as Flan-T5, Flan-UL2, TK-Instruct
- LlamaForCausalLM: such as LLaMA, Alpaca, Vicuna
Evaluation benchmarks include:
- MMLU (Massive Multitask Language Understanding): exam questions covering 57 tasks including mathematics, history, law, medicine, etc., using 5-shot direct prompting, measuring exact-match score.
- BBH (Big Bench Hard): includes 23 challenging tasks on which PaLM (540B) performs below average human raters, using 3-shot direct prompting, measuring exact-match score.
- DROP: a mathematical question answering benchmark, using 3-shot direct prompting, measuring exact-match score.
- HumanEval: includes 164 Python programming problems, using 0-shot direct prompting, measuring pass@1 score.
Related resources The project provides links to the paper, models, and leaderboard. In addition, the project page also mentions the following related work and resources:
- AlgoPuzzleVQA: work on IQ tests for LLMs.
- Resta: work on safety realignment of language models, including paper and GitHub.
- Red-Eval: an LLM safety evaluation benchmark that uses various jailbreak prompts to evaluate safety; it is claimed that on DangerousQA and HarmfulQA benchmarks, it can achieve a 65.1% attack success rate against GPT-4 and a 73% jailbreak success rate against ChatGPT.
- Flacuna: a model obtained by fine-tuning Vicuna-13B on the Flan collection, claimed to outperform Vicuna in problem solving, with the model address at https://huggingface.co/declare-lab/flacuna-13b-v1.0 .
- IMPACT: a dataset for evaluating LLM writing ability, covering four aspects: Informative, Professional, Argumentative, and Creative, downloadable from Huggingface:https://huggingface.co/datasets/declare-lab/InstructEvalImpact .
- Work on FLAN-T5 in text-to-audio generation:https://github.com/declare-lab/tango .
Usage The materials only mention "Install dependencies and download data." and do not provide specific installation commands or usage steps, so detailed usage instructions are not provided for now.
Applicable scenarios Suitable for scenarios that require quantitative comparison of different instruction-tuned models on academic benchmarks, such as researchers and developers evaluating the generalization ability of models on tasks such as MMLU, BBH, DROP, and HumanEval. Compared with existing libraries such as evaluation-harness and HELM, this project emphasizes simple and convenient evaluation of multiple models.
Limitations The materials do not specify the specific link to the model support list, the specific commands for installing dependencies, the detailed steps for running evaluations, hardware requirements, license information, or pricing or charging models, so these are not provided for now.
