llm_multiagent_debate

Purpose

This repository is a preliminary implementation of the paper "Improving Factuality and Reasoning in Language Models through Multiagent Debate," aiming to improve the factuality and reasoning ability of language models through debate among multiple language model instances. The project was accepted at ICML 2024 and was jointly completed by Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch.

Main Capabilities

• Supports running multi-agent debate experiments on multiple tasks, including arithmetic (math), grade school math (GSM), biography, and MMLU.
• Provides generation scripts and evaluation scripts for the corresponding tasks, used to generate answers and evaluate results.
• The repository includes some debate logs and provides links to multi-agent debate implementations on other open-source LLMs.

Usage

The code is divided into different subfolders by task:

• ./math/ contains the code for running the math task.
• ./gsm/ contains the code for running the GSM task.
• ./biography/ contains the code for running the biography task.
• ./mmlu/ contains the code for running the MMLU task.

Specific run commands:

• Math problems: enter the math directory and run python gen_math.py.
• Grade school math (GSM): enter the gsm directory, run python gen_gsm.pyto generate answers, and run python eval_gsm.pyto evaluate results. The GSM dataset needs to be downloaded from the specified link.
• Biography questions: enter the biography directory, run python gen_conversation.pyto generate answers, and run python eval_conversation.pyto evaluate results.
• MMLU: enter the MMLU directory, run python gen_mmlu.pyto generate answers, and run python eval_mmlu.pyto evaluate results. The MMLU dataset needs to be downloaded from the specified link.

When citing the paper, you can use the bibtex file provided by the repository.

Applicable Scenarios

Suitable for researchers and developers studying multi-agent debate mechanisms and the improvement of language model factuality and reasoning ability.

Limitations

This repository is a preliminary implementation. More tasks and settings will be released gradually. Currently, only the code for the above four types of tasks is provided. Specific results, fees, prices, platforms, and other information are not yet provided.