Tests an app the way a human QA pass would, but through several distinct simulated users at
once, each following its own persona's goals, temperament, and quit conditions rather than a
scripted test case. Every run produces evidence-scored judged results and a local report;
nothing here depends on any one target app — what's app-specific lives entirely in the
adapter directory the run selects (adapters/<name>/, contract documented in
adapters/example-http/adapter.md).
Procedure
1. Parse launch intent
Run persona-test.sh with the invocation's arguments (mode positional, --issue, --name,
--n, --target, --adapter, --min-model, --max-model). It validates and prints the
resolved plan line
(mode=... n=... adapter=... target=... issue=... name=... min_model=... max_model=...); exit
0 on success, exit 3 on any malformed input (bad mode, bad flag, --n out of 1–10, a
model outside haiku|sonnet|opus, min_model above max_model, a value-taking flag with no
value), exit 0 with usage text for --help. Read MODE, N, ADAPTER, TARGET, ISSUE, NAME,
MIN_MODEL, MAX_MODEL off that line — these values drive every following step (MIN_MODEL /
MAX_MODEL are none when unset; they bound step 4's per-subagent model choice).
2. Load adapter + personas
Read adapters/<ADAPTER>/adapter.md for its three required hooks (reach/interact, done,
context) and any declared optional overrides (judge dimensions, pass threshold, sink). Load
the starter persona library at personas/*.md.
If N is greater than the starter library's count, generate the shortfall: follow
lib/generate-personas.md, giving it the adapter's context hook text and
M = N - <starter count>. It returns exactly M blocks, each delimited by a line
### FILE: personas/generated/<slug>.md followed by that persona's four-field body — write
each block's content to that exact path before use. If N is at or below the starter count,
use the first N starter personas as-is and skip generation entirely.
3. Build the test plan
Per MODE:
changes— inspectgit diff(or the current branch's PR diff, if one exists against its base) to see what actually changed, and derive one or more scenario names from the touched surfaces — the plan is "exercise what changed", not a fixed script.issue— fetch issueISSUE's body and acceptance criteria (e.g.gh issue view <ISSUE>); the plan is those criteria, one scenario per distinct criterion when there are several.scenario— a single named scenario,NAME, exactly as given — no inference, the user supplied the path to test.
Each (persona × scenario) pair the plan yields becomes one dispatched run.
4. Fan out persona subagents
Dispatch one subagent per (persona, scenario) pair per superpowers:dispatching-parallel-agents.
Each subagent gets: its persona file's four fields (Goal, Temperament, Constraints,
Interaction tendencies), the adapter's reach/interact and done hooks, and the scenario. It
role-plays that persona through the app via the reach hook, stopping per the done hook, and
returns one JSON run record. Validate every returned record immediately against
lib/schema.py's validate_run_record — a record that fails validation is dropped from
the run (never retried, never patched to fit) and counted separately in the final summary.
Pick each subagent's model by how hard its (persona, scenario) is to role-play, then clamp
to [MIN_MODEL, MAX_MODEL]. Default judgment (both none): a happy-path persona whose job
is to walk the obvious flow runs on a cheaper tier (haiku/sonnet); an adversarial or
detail-sensitive persona — the skeptic probing consequences, the power-user hunting silently
dropped params, the accessibility-first reader judging whether output survives being read
aloud — warrants a stronger tier (sonnet/opus) because the finding quality depends on it
noticing subtle failures. Then apply the bounds: MIN_MODEL is a floor (never dispatch below
it), MAX_MODEL a ceiling (never above it); when they are equal, every subagent runs that one
tier; when a set bound clamps a choice, use the bound. Pin the resolved model explicitly on
every dispatch — never let a subagent inherit the caller's model. The external codex driver
(below) is a fixed second-model perspective and is not subject to these bounds.
Hard cap: never dispatch more than 10 persona subagents in one run, regardless of how
large N × scenario-count computes to — persona-test.sh already rejects --n above 10, but
this cap also bounds the (persona × scenario) fan-out from step 3, which can multiply past 10
even when N alone is small.
If an external driver (codex) is installed — check once, e.g. command -v codex —
additionally dispatch at most one such driver, running the same role-play against one
representative (persona, scenario) pair, for a second-model perspective. If it is not
installed, drop this step entirely: no error, no placeholder record, and it doesn't count
against the 10-subagent cap.
5. Judge each run
For every run record that passed validation, dispatch a subagent following lib/judge.md's
instructions, bundling rubric.md verbatim unless the adapter declares a judge-dimensions
and/or pass-threshold override, in which case bundle that override instead. Take the judge
subagent's raw JSON and run it through lib/normalize.py's normalize_judged before using it
anywhere downstream — never consume a judge's raw output directly, always through the
normalizer, so a malformed or partial judge response still yields a well-shaped result.
If the loaded adapter declares a judge-dimensions override, call normalize_judged(raw, dimensions=<the adapter's dimensions>) — passing the adapter's own dimension tuple, not the
normalizer's default — so the adapter's scores survive normalization instead of being zeroed
out under dimension keys the judge never populated. If the adapter also declares extra
judged-result fields beyond normalize_judged's fixed output shape (personaId, scenario, scores, verdict, severity, rationale, suspectedIssues), merge those extra fields from the raw
judge JSON back onto the normalized result before step 6, so nothing the adapter needs
downstream gets dropped by normalization.
6. Aggregate and report
Feed every normalized judged result through lib/aggregate.py's summarize. Always write
the local report via sinks/local_report.py's write(results, runs_dir, run_id) — this
happens unconditionally, before any adapter sink is attempted, and is never rolled back by
anything that follows. Then, only if the loaded adapter declares a sink, run it: map the
results from step 5 — carrying both the adapter's own-dimension scores and any adapter-declared
extra fields — through the adapter's own mapper if it has one, then post with
sinks/http_post.py. A sink failure (network error, missing route, bad auth) does not
invalidate or roll back the already-written local report — surface the failure alongside the
summary, do not fail the whole /persona-test run over it.
7. Report to the user
Summarize, in the final reply: total / passed / failed / worst_severity (straight from
aggregate.summarize's output), how many records were dropped for failing schema validation,
the local report's path, and the sink's outcome if one ran (posted / no sink declared /
failed, with reason).
Graceful degradation rules
- External driver absent — no
codexonPATH→ drop that one dispatch; the run proceeds on the subscription persona subagents alone. Never fabricate its result. - A run record fails schema validation — drop that run, count it, keep going; one bad record never blocks the rest of the batch.
- Sink not declared, or declared but fails — the local report from step 6 stands either way; only the reported sink-outcome line changes.
- Fan-out cap — 10 persona subagents, hard, independent of how N and scenario-count combine.