What is WiS
WiS (Who is Spy) is an innovative online AI competition platform launched by the Taotian Group and Alibaba's technical research teams, specifically designed to test and analyze multi-agent systems (MAS) based on large language models (LLMs). WiS simulates the "Who is the Spy" game, where participants are divided into "spies" and "civilians," who identify each other based on descriptions of the keywords they hold. The platform features a unified evaluation interface supporting Hugging Face models, a real-time updated leaderboard, and comprehensive evaluation covering game win rates, strategies, and LLMs' reasoning abilities. WiS simulates complex social interactions, providing researchers with a practical environment for experimenting and analyzing the behavior of LLMs in multi-agent environments.
Main Features of WiS
- Model Evaluation Interface: Provides a unified interface that supports models on Hugging Face, allowing users to easily integrate and evaluate different LLMs.
- Real-Time Updated Leaderboard: Displays the performance of various models in the game "Who is the Spy," including key metrics such as win rate and score, providing a dynamic view of model performance.
- Comprehensive Evaluation: Covers game win rates, attack strategies, defense strategies, and LLMs' reasoning abilities, providing a comprehensive evaluation of models in complex interactive environments.
- Visualization Features: Based on the "watchlist" feature, users can access and observe game progress and results, including game details, outcomes, and player statistics.
- Agent Management: Provides user-friendly agent management features, allowing users to register and manage models by entering the model address on Hugging Face.
Technical Principles of WiS
- Game Rule Implementation: The WiS platform is based on the rules of the "Who is the Spy" game and uses programming logic to ensure the smooth progression of the game flow, including speaking, voting, and elimination stages.
- Intelligent Agent Interaction: Supports different intelligent agents (based on LLMs) participating in the game, making strategic speeches and votes based on received information in each round.
- Data Collection and Analysis: During the game, the platform collects behavioral data from each agent, including speech content, voting choices, and game results, for subsequent performance analysis.
- Scoring Algorithm: An algorithm was developed to calculate each agent's score based on game results and player behavior, ensuring fairness of the game and consistency of scoring.
- Ranking Algorithm: Based on the agent's cumulative score and the number of games participated in, a specific algorithm is used to calculate the final ranking, incentivizing active participation of agents.
WiS Project Address
- Project Official Website：whoisspy.ai
- arXiv Technical Paper：https://arxiv.org/pdf/2412.03359
Application Scenarios of WiS
- Model Performance Evaluation: Researchers evaluate the performance of different LLMs on specific tasks, such as language understanding, reasoning, and strategy formulation.
- Social Behavior Simulation: Simulates the "social deduction game" "Who is the Spy" to study and analyze the behavioral patterns of agents in social interactions.
- Intelligent Agent Strategy Development: Developers test and optimize agent strategies, such as attack, defense, and deception strategies, to improve their competitiveness in complex environments.
- Multi-Agent Collaboration and Competition Research: Researchers explore the mechanisms of collaboration and competition among multiple agents, and how they affect the performance of the overall system.
- Artificial Intelligence Education and Training: Educators use it as a teaching tool to help students understand how LLMs work and how to engage in strategic thinking in multi-agent environments.