SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

LLM Council - Karpathy's open-source multi-model collaboration framework

Author: AI小集 · Original source: AI工具集

What is LLM Council

LLM Council is Andrej Karpathy's open-sourcemulti-model collaboration framework, which uses OpenRouter to simultaneously distribute a user's question to multiple large models (such as Claude, GPT, Gemini, etc.) for independent answers, with each model anonymously reviewing and scoring the others, and finally a designated Chairman LLM synthesizes all viewpoints to output the optimal answer, achieving multi-model cross-validation and collective decision-making.

LLM Council

Main features of LLM Council

  • Multi-model parallel answering: A single question simultaneously triggers multiple LLMs to generate answers independently, with tabbed views to see each model's original response.
  • Anonymous cross-review: Each model anonymously reviews the other models' answers, ranking them by accuracy and insight, avoiding favoritism toward itself.
  • Chairman model synthesis: A designated Chairman LLM aggregates all drafts and review opinions to output the final unified answer.
  • Local session management: Conversation history is stored locally as JSON files, supporting review and continuation of past discussions.
  • Visualized three-stage process: The web interface clearly displays the complete "draft -> review -> final" pipeline.

Technical principles of LLM Council

  • Backend architecture: Built on FastAPI (Python 3.10+), using asynchronous httpx to concurrently call the OpenRouter API for parallel multi-model requests.
  • Frontend technology: Supports the React + Vite tech stack, with react-markdown to render the Markdown content output by the models.
  • Anonymous review mechanism: In Stage 2, each model's identity is anonymized to ensure the review process is objective and fair, eliminating brand bias.
  • Three-stage pipeline: Stage 1 collects each model's draft -> Stage 2 distributes anonymous answers for mutual review and ranking -> Stage 3 the chairman model synthesizes the final draft.
  • Local data storage: Session data is saved as JSON files in the data/conversations/ directory, with no external database required.

How to use LLM Council

  • Clone the repository: Run git clone https://github.com/karpathy/llm-council.git to download the project code.
  • Install dependencies: For the backend, run uv sync; for the frontend, enter the frontend directory and run npm install。
  • Configure the API key: Create a .env file in the project root directory and fill in OPENROUTER_API_KEY=sk-or-v1-...。
  • Customize models(optional): Edit backend/config.py to modify the COUNCIL_MODELS list and CHAIRMAN_MODEL。
  • Start the application: Run ./start.sh for one-click startup, or start the backend and frontend development servers separately.
  • Start a conversation: Visit http://localhost:5173in your browser, enter a question to trigger the multi-model advisory council process.

Core advantages of LLM Council

  • Eliminates single-model bias: Through multi-model cross-validation and mutual criticism, it significantly reduces hallucinations, blind spots, and inherent bias.
  • Significantly improved answer quality: Through the dual mechanism of review-based filtering + chairman synthesis, the final output is more comprehensive and reliable than a single model.
  • Fully local deployment: Code is open source, data is stored locally, and conversation privacy does not depend on any third-party platform.
  • Highly flexible model combinations: Through OpenRouter, any models can be freely combined, and the community has expanded support for Ollama local model integration.
  • Lightweight and fast setup: 99% of coding is AI-assisted (Vibe Code), with minimal dependencies, and local deployment can be completed in just a few minutes.

LLM Council's project address

  • GitHub repository：https://github.com/karpathy/llm-council

Comparison of LLM Council with similar competing products

Dimension LLM Council AutoGen CrewAI
Product positioning Multi-model answer integration and review tool Conversational multi-agent orchestration framework Role-driven task collaboration framework
Collaboration mode Parallel answering + anonymous peer review + chairman arbitration Group chat conversation + negotiated consensus Role division + sequential/hierarchical execution
Model source Cross-vendor (OpenRouter aggregation) Single or homogeneous models Single or homogeneous models
Anonymous review ✅ Native support ❌ Not supported ❌ Not supported
Peer review scoring ✅ Native support ❌ Not supported ❌ Not supported
Chairman summary ✅ Native support ❌ Requires custom implementation ❌ Can be replaced by a manager Agent
Deployment method Fully local Local/cloud Local/cloud
Data privacy Data is stored completely locally Depends on deployment method Depends on deployment method
Learning curve Extremely low (one-click startup) Medium (requires understanding the conversation paradigm) Low (role configuration is intuitive)
Applicable scenarios Q&A decision-making, content review, model evaluation Code generation, scientific research exploration, debate Content creation, market research, process automation

LLM Council's application scenarios

  • Complex problem decision-making: Important decision-making scenarios such as technology selection and business strategy that require cross-validation of multiple perspectives.
  • Academic research assistance: When reading papers or books, obtain different interpretations and in-depth analyses of the same concept from multiple models.
  • Content quality review: Let multiple models review articles, code, or proposals for each other, automatically selecting the best version.
  • Horizontal evaluation of model capabilities: Compare the answer quality of different LLMs on the same question to assist teams in model selection.
  • Creative brainstorming: Use multi-model perspectives to inspire more diverse creative solutions and avoid single-minded thinking patterns.