SuperAPI AI Directory
Partner中文AI newsFavorites
Skillsfailure-flywheel

failure-flywheel

General & other

View repository

Updated 10-15 22:33

About this skill

Skill: failure-flywheel

Turn recurring failures into rules. Capture by code, count deterministically, draft a stub for a human to approve — never auto-promote.

WHAT

Purpose and context

  • Purpose: turn recurring agent failures into guardrails: record failures in code, count deterministically by taxonomy, and when a failure class crosses the threshold, build a STUB proposing a rule/skill into the propose→gate flow.
  • Trigger (when to use):
    • A task failed in a way that could recur (invented an API, broke a test, ignored a rule, over-built). Record it so it is counted, not forgotten.
    • Periodically (end of session, or every N failures): --report to see which failure class recurs most.
    • When a class crosses the recurrence threshold: --draft <category> to seed a candidate rule, then run /propose to finish it.
    • User says "log this failure", "record the mistake", "why does this keep happening", "turn failures into rules", "failure taxonomy", "what keeps breaking", or /failure-flywheel.
  • Non-goals: Do NOT use to auto-write rules. This skill seeds the gate; a human always approves. Do not distill the rule content yourself (the 10% that needs judgment belongs to the human + /propose).

Mental model

failure → record(category) → failures.jsonl → --report (group · COUNT · leaderboard) → lớp ≥ recurrence_threshold → --draft stub → /propose → cổng người duyệt. Hamel: create → label → fix → repeat. The machine does the 90% mechanical part; the 10% "what the rule really is" is for the human.

Input and output contract

Field Required? Meaning
In <category> + "<summary>" yes (record) nearest taxonomy class; an unknown category is still recorded, flagged in the report
In --detail "..." no extra details
In --date YYYY-MM-DD no (draft) ISO only, default 2026-06-29; wrong format → exit 2
In harness/failure-flywheel.config.yaml yes threshold + taxonomy (adapter, verified: false)
Out line in failures.jsonl yes (record) local history, gitignored
Out leaderboard yes (report) count by class, first/last seen, which class DRAFT
Out llmwiki/wiki/sources/draft/DDMMYY-failure-<category>.md only when the threshold is met STUB (OKF frontmatter + ## Origin, no ## Plan) — stops and waits for a human

Rules and capabilities

  • RULE-01 (MUST): Capture is fail-open — recording a failure must never break the session.
  • RULE-02 (MUST): Never auto-promote. --draft seeds /propose; a human approves.
  • RULE-03 (MUST): The draft filename date is fixed (--date, default 2026-06-29, ISO YYYY-MM-DD only — not the DDMMYY wiki filename convention) for determinism; real use can stamp today. A malformed --date now exits 2 with a clear error instead of silently falling back to the default (fixed 2026-07-20 after that exact silent-fallback shipped a wrong-dated draft file).
  • RULE-04 (MUST): Do not hard-code the threshold or taxonomy anywhere but the config — that is the adapter.
  • Capabilities: write local error history; read threshold/taxonomy config; write one draft stub file into the wiki draft. No network, no editing real rules/skills.

Failure boundaries

  • Writing a broken error → fail-open (session does not break), no blocking.
  • --draft class has not yet met the threshold → do not write a stub (blocked narrow, report current count).
  • --date malformed → failed exit 2, clear message.
  • Config still has verified: false → stub has only human-TODO, no self-written rules (partial intentional).

HOW

Main workflow

Step Type Inputs Action Outputs/exit Failure/next
W01 effect failure just occurred record <category> "<summary>" under the nearest taxonomy class jsonl line unknown category → still record + flag in report
W02 deterministic failures.jsonl + config --report read leaderboard class marked DRAFT if ≥ threshold no class qualifies → stop
W03 effect class meets threshold --draft <category> build stub file draft stub threshold not met / --date wrong → stop
W04 judgment stub STOP wait for human; human runs /propose to become a gated rule/skill full proposal human declines → keep stub

Details of each step (source of truth for W01–W04):

The flywheel (Hamel: create -> label -> fix -> repeat)

The slow human step is "notice the same mistake keeps happening and turn it into a rule." This makes the mechanical 90% deterministic (capture + taxonomy + count + scaffold) and refuses to fake the 10% that needs judgement (what the rule actually is).

Commands

# 1. CAPTURE — append one failure (BY CODE; failures.jsonl is gitignored local history)
python3 harness/scripts/failure-flywheel.py record <category> "<summary>" [--detail "..."]

# 2. REPORT — deterministic taxonomy: group by category, COUNT, leaderboard (most-frequent
#    first) + first/last seen + which classes are eligible to draft
python3 harness/scripts/failure-flywheel.py --report

# 3. DRAFT — only if <category> recurs >= recurrence_threshold: scaffold a STUB proposal at
#    llmwiki/wiki/sources/draft/DDMMYY-failure-<category>.md and STOP for human approval
python3 harness/scripts/failure-flywheel.py --draft <category> [--date YYYY-MM-DD]

Steps

  1. On a failure, record it under the closest taxonomy category (see the config). Capture never blocks — an unlisted category is still recorded and flagged in --report.
  2. Run --report to read the leaderboard. A class marked DRAFT has recurred >= the threshold.
  3. For an eligible class, --draft <category>. It writes a valid draft STUB (OKF frontmatter + ## Origin, deliberately no ## Plan so it passes the R7 gate as a seed) containing a templated "TODO: distill rule from these N failures" + the failure list.
  4. The stub STOPS for you. Run /propose to turn it into a complete, gated rule/skill. FailureFlywheel never auto-promotes — the gate stays human.

Package into instructions for next time (user 23/09/2026)

Error class reaches recurrence_threshold (default 3 — i.e. more than 2 times) then record() AUTO-packages immediately into an instruction card:

harness/metrics/guardrails/failure-<lớp>.md     # máy gói, có INDEX.md; được commit để phiên/máy khác cùng học
python3 harness/scripts/flywheel.py --kind failure --guardrails   # gói lại toàn bộ lớp đã đủ ngưỡng

The card only DISTILLS from the very lines already recorded (symptom + fix used + count + most recent) — no inventing new rules. Hook session_start inserts the 5 newest cards at the start of EVERY session (section 🧯 [đã học]), so next time the agent reads them before doing the same kind of work. The card carries status đã học, CHƯA duyệt: it is a read-to-remember instruction, it CANNOT bite in CI. To become an official rule/skill you still --draft then /propose — the human gate stays unchanged (RULE-02).

The adapter boundary (build-now-adapt-later)

Everything above is deterministic and built now. The ONE quarantined unknown is harness/failure-flywheel.config.yaml (verified: false): the recurrence threshold and taxonomy are best-guesses, and the "distill failures -> rule" model is absent. While it is unset, --draft inserts a human-TODO stub instead of an auto-written rule. Finalize later by editing only that one file — calibrate the threshold, name a distill model, flip verified: true.

Branches

ID Kind Guard Behavior Skip / failure Rejoin
B01 conditional_required error class ≥ recurrence_threshold W03 --draft then W04 below threshold → skip, report only W04
B02 capability_optional config verified: true + has distill model stub has an auto-distilled rule instead of human-TODO not yet verified → human-TODO stub (current default) W04

Validation and stopping

Counting and thresholds are decided by failure-flywheel.py (deterministic). Valid stub = OKF frontmatter + ## Origin, intentionally not ## Plan to pass the R7 gate as a seed. Always stops at W04 — never auto-promotes.

Examples

  • Positive: agent fabricates an API for the Nth time (exceeds threshold) → python3 harness/scripts/failure-flywheel.py record invented-api "called nonexistent fetchAll()" → --report shows invented-api marks DRAFT → --draft invented-api writes llmwiki/wiki/sources/draft/DDMMYY-failure-invented-api.md and stops awaiting /propose.
  • Boundary/failure: --draft invented-api --date 19/09/2026 → exit 2 with a clear error, does not write a file with the wrong date.

Related

  • harness/scripts/code-logger.py — same by-code, gitignored-JSONL, fail-open capture pattern.
  • llmwiki/skills/dev-loop/propose.md — the gate this feeds.
  • llmwiki/skills/dev-loop/build-now-adapt-later.md — the quarantine pattern the config follows.