council

Skill: council

🧭 Choose the backend for each seat (cheaper model vs Claude), run multiple seats in parallel (1 worktree/seat — NOT multiple opencode/1 folder) → see orca-dispatch-reference (single source of truth).

A deterministic harness around Andrej Karpathy's llm-council (https://github.com/karpathy/llm-council) three stages — Stage 1 "First Opinions", Stage 2 "Review" (anonymized peer-rank), Stage 3 "Final Response" (chairman) — wired onto orca orchestration.

WHAT

Purpose and context

• Purpose: runs a Karpathy-style 3-stage LLM council on orca orchestration — N seats answer independently, blind (anonymized) cross-scoring, chairman synthesizes — with a deterministic transcript and a mandatory HTML report (Stage 4).
• Trigger (when to use): Hard questions where one model's answer is risky and you want a panel + an audit trail of who-ranked-what, with the favour-your-own bias removed by blinding. For a single quick answer, just ask one model. Invocations: "run a council", "llm-council", "panel of models", "blind peer-rank these answers", "ensemble + chairman", or /council.
• Non-goals: not used for quick single-answer questions; council.py does not call a model; does not choose the model for a seat (that is the adapter's job harness/council.config.yaml); the report adds no new judgments beyond the transcript.

Mental model

Split of labour:

• harness/scripts/council.py owns the DETERMINISTIC protocol and never calls a model: anonymize (strip author → A/B/C by stable sha256 order), mean-rank aggregation, dissent surfacing, the anchor guard (seed-driven, per-judge presentation order — no Math.random), and the json+md transcript.
• The orca orchestration owns the MODEL GENERATIONS: each seat's answer, each judge's ranking, the chairman's synthesis. It dispatches them exactly like any other multi-agent wave (see orca-workflow, orchestration).
• harness/council.config.yaml is the ONE adapter (verified: false): which model fills each seat, the judge models, the chairman. Every value is an # ASSUMPTION (not verified). The engine never branches on these — it only stamps them into the transcript. Finalize the council by editing this one file.

question → Stage 1 answers.json → Stage 2a blind packet (A/B/C + presentation order) → Stage 2b judges.json → Stage 2c transcript (mean-rank, dissent, chairman_brief) → Stage 3 chairman_synthesis → Stage 4 HTML report versioned.

Input and output contract

	Field	Required?	Meaning
In	question	yes	the same question for every seat; --question to show on the report
In	harness/council.config.yaml	yes	seats / judges / chairman (verified: false)
In	roster persona (--case / --profile / --personas)	no	lens viewpoint added to the Stage-1 prompt
In	--seed N	no	seed anchor guard, overrides anchor_seed
Out	answers.json · judges.json	yes	[{"id","author","text"}] · [{"judge","ranking":[...]}]
Out	scratchpad/council-<slug>/council.packet.{json,md} + council.transcript.{json,md}	yes	blind packet; consensus, winner, dissent, chairman_brief, chairman_synthesis
Out	llmwiki/html/council/council-report-NNN-seed<seed>.html	required	versioned report, no overwrite; latest.html pointer

Rules and capabilities

• RULE-01 (MUST): LANGUAGE (mandatory): every seat/judge/chairman MUST answer in fully diacritized Vietnamese; insert this sentence into EVERY Stage-1/Stage-2/Stage-3 prompt.
• RULE-02 (MUST): Run output always under scratchpad/council-<slug>/ — never bare run/ or a repo-root folder.
• RULE-03 (MUST): Blindness, not exclusion, is the guard — the judge only sees labels A/B/C, in their own presentation order.
• RULE-04 (MUST): The roster always installs ≥1 counterweight pair; odd size (3/5) so mean-rank has no tie.
• RULE-05 (MUST): Stage-4 HTML always renders; renderer errors are WARN only, NEVER kill the command rank or transcript.{json,md}.
• RULE-06 (MUST): council.py never calls a model; the engine never branches on config values — only stamps them into the transcript.
• Capabilities: dispatch model generation through the orchestration layer (seat/judge/chairman); compute the protocol deterministically locally; write transcript + HTML files; no network/CDN for the report.

Failure boundaries

• council.py selftest does not exit 0, or orca status --json has no runtime available → blocked, do not run the council.
• Seat answers without diacritics → the report also loses diacritics (faithful render) → restate RULE-01 and re-run that seat.
• Missing judges → rank only emit the blind packet then stop (partial, wait for Stage 2b).
• HTML renderer error → WARN, transcript still exists (partial for Stage 4, re-run render).
• Roster missing a counterweight pair → stderr warning; mistyped case/persona → nearest suggestion, no bare failure.

HOW

Main workflow

Step	Type	Inputs	Action	Outputs/exit	Failure/next
W01	deterministic	repo, runtime	Preconditions: council.py selftest, orca status --json, config has seats/judges/chairman	eligible	missing → blocked
W02	effect	question (+ lens)	Stage 1 — dispatch one worker per seat, collect answers.json	answers.json	seat error/no diacritics → re-run seat
W03	deterministic	answers.json	Stage 2a — council.py prepare	blind packet	—
W04	effect	packet	Stage 2b — dispatch judges with blinded answers in presentation order, collect judges.json	judges.json	—
W05	deterministic	answers + judges	Stage 2c — council.py rank (self-written Stage-4 HTML)	transcript + report	render error → WARN, B02
W06	effect	chairman_brief	Stage 3 — dispatch chairman, paste chairman_synthesis into transcript	transcript complete	—
W07	deterministic	transcript	Stage 4 — council.py render <transcript.json> so the report has synthesis	new versioned report	error → WARN, transcript kept unchanged

Step-by-step details (source of truth for W01–W07):

Preconditions

• python3 harness/scripts/council.py selftest exits 0 (engine is healthy).
• orca status --json shows a running runtime; orchestration enabled.
• Seats/judges/chairman set in harness/council.config.yaml.

Protocol (maps each stage to an orca dispatch)

Stage 1 — First Opinions (orchestration generates)

Dispatch one worker per seat in council.config.yaml, same question to each.

LANGUAGE (mandatory): every seat/judge/chairman MUST answer in Vietnamese with full diacritics (correct spelling, full tone marks + circumflex marks). ABSOLUTELY do not write Vietnamese without diacritics (ASCII). council.py renders the input text faithfully (ensure_ascii=False, no normalize) — if a seat answers without diacritics then the report loses diacritics too. Insert this sentence into EVERY Stage-1/Stage-2/Stage-3 prompt.

orca orchestration task-create --spec "Answer: <question>" --json
orca orchestration dispatch --task <task_id> --to <seat_handle> --inject --json
orca orchestration check --wait --types worker_done --timeout-ms 300000 --json

Collect the replies into answers.json — [{"id","author","text"}, ...], where author is the seat id (the real identity; it gets stripped next).

Stage 2a — blind packet (council.py is deterministic)

python3 harness/scripts/council.py prepare answers.json --config harness/council.config.yaml --out scratchpad/council-<slug>/

<slug> = short name for this run (e.g. release-gate). Always under scratchpad/ (gitignored, per-repo convention) — never bare run/ or a repo-root folder; those leak as untracked clutter that never gets cleaned up (lesson 170726, 6 stray run-council*/ dirs found at repo root from past sessions copying this example literally). Writes scratchpad/council-<slug>/council.packet.{json,md}: the answers relabelled A/B/C with authors removed, plus each judge's presentation order from the anchor guard. Show each judge its answers in its row's order to cancel position bias.

Stage 2b — Review (orchestration generates)

Dispatch each judge in council.config.yaml the BLIND answers, in that judge's presentation order. Ask each to return a ranking of the labels (best first). Collect into judges.json — [{"judge","ranking":["B","A","C"]}, ...].

Blindness, not exclusion, is the guard: a judge may be a seat, but it cannot recognise its own answer, so it cannot play favourites.

Stage 2c — aggregate (council.py is deterministic)

python3 harness/scripts/council.py rank answers.json --judges judges.json --config harness/council.config.yaml --out scratchpad/council-<slug>/

Writes scratchpad/council-<slug>/council.transcript.{json,md}: mean-rank consensus, the winner, the dissent table (most-contested answer), and a chairman_brief.

Stage 3 — Final Response (orchestration generates)

Dispatch the chairman the chairman_brief from the transcript (consensus order + the dissent points it must resolve). Its synthesis is the final answer; paste it back under chairman_synthesis in the transcript for the record, then council.py render <transcript.json> to regenerate the Stage-4 HTML with the synthesis shown (the report auto-written during rank has an empty synthesis because rank rebuilds from answers+judges only).

Stage 4 — HTML report (MANDATORY, self-rendered in council.py, offline)

Always render — no flag needed, no dependency on external skills. Each time rank succeeds, render_report_html(t, personas) (located right inside council.py) writes a .html versioned — llmwiki/html/council/council-report-NNN-seed<seed>.html (NNN increments, NO overwriting → each run is an immutable record) + latest.html is a convenient pointer (gitignore, avoids diff-churn). No CDN (system font + inline SVG favicon/grain) → offline; no coupling docs-site-macos. Feed purely from the just-built transcript (t) → invent nothing new, every seat string html.escape in place (anti HTML-injection). The whole render block is wrapped in try/except: renderer errors only WARN, NEVER kill the command rank or transcript.{json,md} (Taleb blast-radius guard). Council member names+lenses are taken from council.personas.yaml; the question is displayed when passing --question (left blank → the report hides the question line instead of printing a placeholder).

The page has exactly three sections, in order:

1. Council opinions. One card per seat (border color = sha256(author), ordered by consensus rank + medal), answer auto-formatted into (1)(2)(3) as a list. Source: t["answers"] + t["aggregate"].
2. BLIND vote. The vote table uses anonymous labels A/B/C: each judge one row, ranking by label + presentation-order of the anchor guard. Reveal map A/B/C → author only at the end of the section. Source: t["judge_rankings"] + t["anchor_guard"].
3. Final dashboard. KPIs (winner / most-contested / consensus) + mean-rank consensus table, and chairman_synthesis is shown prominently WHEN it has been pasted into the transcript (the rank is rebuilt from answers+judges so this field is empty until the chairman fills it — at which point the synth block auto-hides). Source: t["winner"] / t["most_contested"] / t["aggregate"].

Since all figures come from the already-deterministic transcript, the report is only a presentation layer — adding no new judgment. Same transcript → same HTML.

Branches

ID	Kind	Guard	Behavior	Skip / failure	Rejoin
B01	user_optional	want diversity of PERSPECTIVES (few providers)	draw a persona roster (Persona lenses section below), inject the lens into the Stage-1 prompt (dynamic) or the field persona: in config (fixed); roster + reason written to the transcript	not drawn → seat = plain model	W02
B02	recovery	HTML report errored or need to show chairman_synthesis	re-run council.py render <transcript.json> → new versioned file	renderer still fails → WARN, transcript is the source	end
B03	conditional_required	run rank has no --judges	only emit blind packet then stop	—	W04

Validation and stopping

Deterministic: selftest (12 checks) prove the same answers.json + judges.json + seed → transcript byte-identical; report is only a presentation layer of the transcript. Needs review: chairman's synthesis quality. Stop after W07 when the report has synthesis.

Examples

• Positive: "run a council: should we split the monorepo or not" with roster --case decision 3 seats → 3 seats answer in Vietnamese with diacritics → prepare outputs A/B/C → 3 judges rank → rank --out scratchpad/council-monorepo/ outputs winner + dissent → chairman synthesizes → render writes council-report-NNN-seed<seed>.html has synthesis.
• Boundary/failure: council.py selftest exit non-zero → stop at W01, no seat dispatched. Renderer fails during rank → only WARN, council.transcript.json is still written, rerun render afterwards.

Reference — Persona lenses — "great person" perspectives (optional, ADDITIVE)

By default each seat = one MODEL answers. The persona layer adds diverse PERSPECTIVES: each seat wears a lens (Feynman / Munger / Taleb / Rams …) — important when model diversity is limited (only a few providers). The engine council.py does NOT change: persona is just text inserted into the Stage-1 prompt.

Forgot a name / syntax? council.py roster --list (or roster empty) prints all cases (by TASK) · profile · 18 personas — choose --case by task, no need to remember names. Misspell it and it still calls correctly: matches by NAME and id, case-insensitive (--personas Feynman,Taleb ok); typo → closest suggestion (--case risks → "did you mean 'risk'?"), never fails cold.

Pick 3-5 people by case (pure code, loggable):

python3 harness/scripts/council.py roster --list                 # catalog: case/profile/persona
python3 harness/scripts/council.py roster --case risk            # 3 ghế, có ≥1 cặp đối-trọng
python3 harness/scripts/council.py roster --case ml-ai --size 5  # 5 ghế
python3 harness/scripts/council.py roster --profile lean         # 5 người execution-lean
python3 harness/scripts/council.py roster --personas feynman,taleb,rams --json

• Case tag: design · strategy · debug · risk · product · decision · simplify · ml-ai (table in harness/council.personas.yaml).
• Rules: roster always installs ≥1 opposing pair (prevents echo chamber); if missing → warning on stderr. Odd size (3/5) so mean-rank has no ties.
• Library: 18 personas + 13 opposing pairs (source github.com/0xNyk/council-of-high-intelligence).

Used in the protocol: after drawing the roster, assign each persona to a seat — there are 2 ways:

1. Dynamic (recommended): orchestrator takes the output roster --json, for each seat inserts the lens into the Stage-1 prompt: "Answer through the lens of <name>: <lens>. <sig>."
2. Fixed: fill the field persona: for each seat in harness/council.config.yaml.

Roster + reason for drawing (case, tension pairs) write into transcript so it is auditable (Pillar 5). The "which model fills the seat" part remains unknown, quarantined in council.config.yaml (verified) — the persona-lens is independent of it.

Reference — council.py commands

Command	Does
rank <answers.json> --judges <judges.json>	full aggregation → transcript.json + .md
rank <answers.json> --judges <j>	+ auto-writes versioned llmwiki/html/council/council-report-NNN-seed<seed>.html (Stage-4, mandatory)
rank <answers.json> (no judges)	emits the blind packet, then stops
prepare <answers.json>	blind packet only (Stage 2a)
render <transcript.json>	re-render Stage-4 HTML from the existing transcript (e.g. after pasting chairman_synthesis) → new versioned file
roster --case <tag> / --profile <p> / --personas a,b,c	picks 3-5 persona-lenses (pure lookup, ≥1 opposing pair); --size 3|5, --json
selftest	conformance vectors; asserts determinism + correctness

Flags: --seed N (anchor-guard seed; overrides config anchor_seed), --out DIR (transcript json/md), --config harness/council.config.yaml, --question "..." (the real question shown on the report; left blank → question line hidden). Stage-4 HTML versioned is always written to llmwiki/html/council/ — no flag needed.

Reference — Adapter boundary (build-now-adapt-later)

• Contract (built + tested now): the json schemas (answers.json, judges.json, transcript) and all the deterministic ops in council.py.
• Quarantine (verified: false): model identities in harness/council.config.yaml. The math is independent of them.
• Adapt later (one file): edit seats/judges/chairman in the config, run a real council, then flip verified: true. No engine change.

Reference — Determinism guarantees

Same answers.json + judges.json + seed → byte-identical transcript every run. Anonymization depends on sha256(id), not input order or author, so neither position nor authorship leaks. The anchor guard is seeded per judge from the arg — never the global RNG. selftest proves all of this (12 checks).