What is AgentReview
AgentReview is a framework based on large language models (LLMs) that simulates the academic peer review process. AgentReview uses LLM agents to simulate the roles of reviewers, authors, and area chairs, helping researchers explore the impact of review bias, roles, and decision mechanisms on review outcomes while respecting privacy. AgentReview can provide insights for improving peer review mechanisms and support future research.
Main Features of AgentReview
- Simulating the peer review process: AgentReview simulates the real academic peer review workflow, including stages such as reviewer evaluation, author response, reviewer discussion, and area chair decision-making.
- Role simulation: The framework integrates three roles: reviewers, authors, and area chairs (AC), each driven by LLM agents and exhibiting different behavioral characteristics.
- Multivariable analysis: Based on simulation, AgentReview explores and isolates multiple variables that affect review outcomes, such as reviewers' commitment, intent, and knowledge ability, as well as the AC's decision style.
- Privacy protection: During the simulation process, AgentReview respects the privacy of review data and does not require real sensitive review data.
- Validation of sociological theories: AgentReview validates the application of sociological theories such as social influence theory, altruism fatigue, groupthink, and authority bias in peer review.
Technical Principles of AgentReview
- Large language models (LLMs): AgentReview is built on LLMs and uses language understanding and generation capabilities to simulate the behavior of reviewers and authors.
- Agent modeling: Each role in the framework (reviewer, author, AC) is modeled as an agent with specific attributes and behaviors, and agents interact according to preset characteristics and rules.
- Structured review process: AgentReview follows a structured five-stage review process, simulating the entire process from initial review to final decision.
- Customization and extensibility: The framework is designed to be extensible, supporting researchers in customizing role attributes and review processes as needed.
- Data-driven insights: Based on data generated from large-scale simulations, AgentReview provides statistically significant insights and supports content and numerical analysis.
AgentReview Project Addresses
- Project Website：agentreview.github.io
- GitHub Repository：https://github.com/Ahren09/AgentReview
- HuggingFace Model Library：https://huggingface.co/spaces/Ahren09/AgentReview
- arXiv Technical Paper：https://arxiv.org/pdf/2406.12708
- Online Experience Demo：https://huggingface.co/spaces/Ahren09/AgentReview
Application Scenarios of AgentReview
- Academic journal and conference management: Used to optimize and manage the peer review process for academic papers, improving review quality and efficiency.
- Reviewer training and development: Serves as an educational tool to help new reviewers learn review standards and best practices.
- Sociological and psychological research: Based on simulation, validates the application of theories such as social influence and groupthink in review behavior.
- Interdisciplinary review research: Compares review standards and processes across different disciplines, providing a basis for designing interdisciplinary journals.
- Policy making and evaluation: Assists policymakers in evaluating and formulating policies and guidelines related to peer review.