llm-council

Purpose

The core idea of LLM Council is: instead of asking only a single LLM, organize multiple models (such as OpenAI GPT, Google Gemini, Anthropic Claude, xAI Grok, etc.) into an "LLM council," letting them each answer first, then review and rank each other, and finally synthesize a final answer. It is suited for users who want to compare the output quality of multiple models side by side and observe cross-opinions among models.

Main capabilities

After submitting a query, the process is divided into three stages:

• Stage 1: Initial opinions. The user query is sent separately to each LLM in the council, their respective answers are collected, and displayed in a tabbed view for easy individual review.
• Stage 2: Review. Each LLM receives the answers of the other LLMs. The underlying system anonymizes model identities (Response A/B/C/...), to avoid favoring one's own model when judging; models are asked to rank these answers by accuracy and insight.
• Stage 3: Final answer. The designated Chairman model synthesizes the answers of all models and compiles them into a final answer presented to the user.

This fork version adds the following capabilities on this basis:

• Conversation history persistence: load history records from the backend data/conversations.json, and remain compatible with the legacy data/ layout (including JSON files split by conversation under data/conversations/), so that earlier run records reappear in the interface.
• Full sidebar list: the frontend requests /api/conversations?limit=500, to avoid history records being truncated too early.
• Stage 2 judge deduplication: deduplicate judge models, to avoid the same judge being counted multiple times.
• Title persistence: after Stage 3 completes, generate a title and save it to the conversation record, so it remains after refresh and appears in history.
• Auditable decisions: each council answer saves a concise decision trace (Stage 1 answers, Stage 2 rankings, Stage 2 to model mapping), making it easy to inspect afterward why the Chairman reached that conclusion.
• Reproducible rankings and scores: persist aggregate rankings (average rank + count), making it easy to compare council behavior across different runs and different prompts.
• Tools and reproducibility: add Stage 2 smoke/quality scripts and accompanying evaluation artifacts, making regression testing repeatable.
• Role settings: the role set in the implementation is Builder / Reviewer / Synthesizer / Contrarian, and provider default mappings are provided in backend/roles.py; earlier fork descriptions once described them as Analyst / Researcher / Critic / Provocateur.
• Real SSE endpoint: /api/conversations/{id}/message/stream uses text/event-stream, progressively pushing stage*_start/complete events.
• Persist meta/metadata (label_to_model, aggregate_rankings, model_roles), so that Stage 2 renders correctly and runs can be reviewed afterward.
• The frontend Stage 2 reads msg.meta || msg.metadata, compatible with both the fork and upstream payload structures.

How to use

1. Install dependencies: the project uses uv for project management.
2. Configure API Key: create a .env file in the project root directory, obtain an API key at openrouter.ai, and purchase the required credits or enable auto-recharge.
3. Configure models (optional): edit backend/config.py to customize council members.
4. Run the application: you can use the startup script, or run manually—one terminal for the backend, one terminal for the frontend, then open in the browser http://localhost:5173。

Tech stack: the backend is FastAPI (Python 3.10+), async httpx, OpenRouter API; the frontend is React + Vite, rendered using react-markdown; storage is JSON files under data/conversations/; package management uses uv for Python and npm for JavaScript.

Use cases

• Compare the answers of multiple LLMs to the same question side by side.
• Observe cross-evaluations and rankings among multiple models of each other's outputs.
• Research or experimental purposes that require retaining historical run records and reviewing the decision-making process.

Limitations

• The project is described by its author as a 99% vibe coded weekend experiment; the original author explicitly states that no support is provided, there is no intention to improve it, and the code is for inspiration only.
• You need to prepare your own OpenRouter API key and purchase credits.
• Specific costs, performance, and results are not provided in the materials.