fan-out-work

Fan-Out Work

Inputs

$work: the work-list request — a description of the same treatment to apply to N items, or a path to an existing item list.

Overview

Turn "do X to every item in a list" into a controlled parallel run: discover the list, map each item through its stages concurrently with caps and isolation, optionally reduce to a merged result, and report faithfully — including what was dropped. The win is wall-clock (the slowest single item's chain, not the sum), without the usual parallel sins: silent truncation, hidden failures, and write races.

On Claude Code, that wall-clock win is real only when the session runs at the ultracode effort level (the mode that makes the Workflow tool / concurrent agents the default) — a runtime harness setting, distinct from this skill's static effort: high frontmatter. Without it the map runs sequentially: every item is still covered with the same results, isolation, and honest accounting — only slower. A sequential run is NOT a truncation (rule 1), so never report it as reduced coverage. If throughput matters and ultracode is off, tell the user they can enable it (/effort ultracode) — then proceed.

Phase 0: Scout the work-list inline (before any fan-out)

Discover the items in the coordinator FIRST — you usually can't know the fan-out's shape until you've listed the work:

enumerate the items (files, modules, call sites, findings) with a real command (git diff --name-only, grep -rl, a glob) — not a guess;
record each item's identity so results can be attributed and duplicates removed;
prove independence: no item consumes another's output; write scopes are disjoint (or will be isolated). If they're not independent, STOP — this is a DAG, not a fan-out.

Success criteria: a concrete, de-duplicated item list, each item verified independent (or isolatable).

Phase 1: Define the per-item pipeline

Specify the stages each item flows through — the SAME stages for every item, run per-item independently:

e.g. find gap → write test → mutation-verify (auto-test), or review → adversarially verify → fix (auto-review), or transform → validate (a migration);
default to a pipeline (no barrier between stages): item A can be at stage 3 while item B is still at stage 1 — wall-clock is the slowest single chain, not the slowest stage summed. Only use a barrier (collect all of stage N before stage N+1) when a stage genuinely needs the whole set — dedup/merge across items, or an early-exit on total count.

Success criteria: the per-item stage sequence is defined; pipeline-vs-barrier chosen with a reason.

Phase 2: Set caps, isolation, and the drop policy

Before launching, pin the safety rails:

Concurrency cap — how many items run at once (the runtime caps at ~min(16, cores−2); set a tighter cap for heavy/worktree items to avoid rate limits and disk pressure). Excess items queue and still run — a cap bounds simultaneity, not total.
Isolation — writers get isolation: 'worktree'; read-only items don't need it.
Retry — decide whether a failed/rate-limited item retries (bounded backoff) or is recorded blocked.
Drop policy — if you are deliberately bounding coverage (top-N by risk, sampling), decide the rule NOW and plan to LOG it. If you intend full coverage, there is no drop — every item runs.

Success criteria: concurrency, isolation, retry, and (explicit) drop policy are all decided.

Phase 3: Run the fan-out (Workflow tool)

Author a Workflow that maps the items through the pipeline. The script owns the loop, caps, and accounting; the agents do the per-item work. Filter dead items with .filter(Boolean) and keep their identities for the report. See references/fanout-patterns.md for the pipeline / barrier / loop-until-dry shapes and the map-reduce recipe.

For a modest list where a full Workflow is overkill, a single-message batch of Agent(...) calls (parallel, backgrounded, isolated for writers) is a lighter equivalent — same rules apply.

Success criteria: every item ran (or is a tracked null); the coordinator has a result per item.

Phase 4: Reduce and report faithfully

Aggregate the per-item results, then report the WHOLE truth:

covered: items that completed, with their outcomes;
failed/blocked: items that returned null / errored — named, with the reason;
dropped (if a cap was intentional): the exact items not attempted and the rule that excluded them — never omitted so the run looks complete.

If a reduce stage merges results (dedup findings, combine a report), do it over the full covered set.

Success criteria: the report accounts for every item — covered, failed, or dropped — with no silent gaps between N discovered and N reported.

Common Rationalizations

Rationalization	Reality
"I did the first 20, that's representative — call it done."	A sample is not full coverage. If you capped, say "20 of 200, by risk"; don't report success.
"That item errored, I'll leave it out of the summary."	A dropped failure is a hidden failure. It belongs in the report as blocked, with the reason.
"They probably don't conflict, just run them in parallel."	"Probably" is how two writers corrupt a file. Prove independence or isolate; don't hope.
"A barrier is cleaner to code."	A barrier makes fast items wait for the slowest at every stage. Use a pipeline unless a stage needs the whole set.
"One big rewrite across all files is simpler than N items."	It also fails atomically and hides which item broke. Per-item is debuggable and partially recoverable.

Red Flags

The count of items reported is smaller than the count discovered, with no "dropped/blocked" note.
Parallel agents writing overlapping paths without worktree isolation.
A summary that says "all done" after a run that hit a concurrency/no-retry cap.
Fanning out items where one clearly needs another's output (should be a DAG).
.filter(Boolean) results used without noting how many were filtered.

Guardrails

Never silently truncate — log every intentional cap and every dropped/blocked item.
Never hide a failed item to keep the summary clean.
Never fan out writers without isolation, or dependent items as if independent.
Never move the continue/stop decision into the agents — the coordinator owns termination.
Prefer pipeline over barrier; reach for a barrier only when a stage needs the full set.

When To Load References

references/fanout-patterns.md The Workflow-tool shapes — pipeline (no barrier), barrier (dedup/early-exit), map-reduce, per-item loop-until-dry, and the single-message Agent-batch equivalent — plus the concurrency-cap and faithful-aggregation recipes. Load when authoring the fan-out.

Output Contract

Report:

items discovered (count + how enumerated) and independence basis
per-item pipeline + pipeline/barrier choice; concurrency, isolation, retry, drop policy
covered (outcomes), failed/blocked (named + reason), dropped (named + rule) — accounting for every item
any reduced/merged result