What is Optima
Optima is a framework launched by Tsinghua University to optimize large language model (LLM)-based multi-agent systems (MAS). Based on an iterative generation, ranking, selection, and training paradigm, it significantly improves communication efficiency and task effectiveness. Optima balances task performance, token efficiency, and communication readability, explores multiple reinforcement learning algorithms, and integrates Monte Carlo Tree Search technology to generate high-quality training data. In multi-agent tasks, Optima demonstrates performance surpassing single-agent baselines and traditional MAS, achieving up to a 2.8x performance improvement and reducing token usage. Optima's efficiency improvements offer new possibilities for more effective inference computation and improved inference-time scaling laws.
Optima's Main Features
- Improved Communication Efficiency: O optimizes inter-agent communication in multi-agent systems (MAS), reducing the number of tokens required to complete tasks and improving communication efficiency.
- Enhanced Task Performance: Based on iterative training and a balance of reward functions, it improves agents' performance in complex tasks, including information-asymmetric question answering and complex reasoning tasks.
- Scalability: It supports MAS in maintaining effectiveness when handling larger-scale and more complex tasks, improving the system's scalability.
- Improvement of Inference-Time Scaling Laws: By reducing token usage, it provides possibilities for improving inference-time scaling laws, helping achieve better performance at lower computational cost.
Optima's Technical Principles
- Iterative Training Paradigm: Based on an iterative generate, rank, select, and train paradigm, it gradually optimizes agent behavior.
- Reward Function: A reward function is designed to balance task performance, token efficiency, and communication readability, guiding agents to complete tasks while maintaining communication efficiency.
- Reinforcement Learning Algorithms: It explores multiple reinforcement learning algorithms, including supervised fine-tuning (SFT), direct preference optimization (DPO), and hybrid methods, to optimize agent behavior.
- Monte Carlo Tree Search (MCTS): It integrates MCTS heuristic techniques, treating dialogue turns as tree nodes, exploring diverse interaction paths, and generating high-quality DPO training data.
- Multi-Objective Optimization: Based on the reward function, it simultaneously considers multiple objectives, focusing on communication efficiency and output interpretability while improving task performance.
Optima's Project Address
- Project Official Website：chenweize1998.github.io/optima-project-page
- GitHub Repository：https://github.com/thunlp/Optima
- arXiv Technical Paper：https://arxiv.org/pdf/2410.08115
Optima's Application Scenarios
- Information-Asymmetric Question Answering: In question-answering systems, when the answer to a question requires integrating information from multiple sources, optimize communication among agents to improve answer accuracy and response speed.
- Complex Reasoning Tasks: For problems requiring multi-step reasoning, such as legal case analysis and scientific problem solving, it helps agents collaborate more effectively and reach correct conclusions.
- Software Development: In software development, coordinate the development of different functional modules, optimize the development process, and improve code quality based on effective communication among agents.
- Decision Support Systems: In business decision-making or policy formulation, help multiple decision-makers or agents share information, discuss options, and reach consensus.
- Multi-agent games: In games that require multiple players or agents to collaborate, optimize communication strategies among players and improve team cooperation efficiency.