SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillscontext-audit

context-audit

Coding & development

View repository

Updated 10-15 23:32

About this skill

Context Audit

Audits the project's three-layer context management system (CLAUDE.md / rules / knowledge) against the progressive disclosure architecture:

CLAUDE.md (always loaded) — describes architecture, no detailed index
  └─ Rules (.claude/rules/) (always loaded) — inline `> 详见` refs to knowledge
       └─ Knowledge (docs/knowledge/) — on-demand deep experience

When to Use

  • After adding new knowledge files — verify they're indexed from a rule
  • After refactoring rules — verify no references broke
  • Periodic hygiene check — catch drift over time
  • After onboarding a new domain — verify the new rule covers its knowledge files

Execution

Phase 1: Discovery

Scan the project to find all context files:

# Knowledge files
find docs/knowledge/ -name "*.md" -type f 2>/dev/null | sort

# Rule files
find .claude/rules/ -name "*.md" -type f 2>/dev/null | sort

# CLAUDE.md
ls CLAUDE.md 2>/dev/null

If docs/knowledge/ or .claude/rules/ doesn't exist, report and skip the corresponding checks.

Phase 2: Cross-Reference Audit

For each knowledge file found in Phase 1:

  1. Grep rule files first for references to that knowledge file (match docs/knowledge/<filename> or bare <filename>)
  2. If not found in rules, grep knowledge index files (docs/knowledge/*-index.md or files whose name ends with -index.md). An index file that is itself rule-referenced transitively covers the leaves it lists.
  3. Record which rule(s) and/or index reference it
  4. Mark files as:
    • Directly covered: referenced from at least one rule
    • Indirectly covered: referenced from a knowledge index that is itself rule-covered (legal two-tier pattern)
    • Orphaned: not reachable from any rule, even transitively (action needed)
    • Deprecated: contains [DEPRECATED] marker (info only, no action needed)

For each reference found in rule/AGENTS/CLAUDE files:

  1. Verify target exists: check the referenced knowledge file actually exists on disk
  2. Skip placeholder examples: refs that look like syntax illustrations (e.g., docs/knowledge/xxx.md, <filename>.md, foo.md) inside example/format sections are not real refs. Detect by:
    • filename matches xxx, foo, bar, <.*>, or
    • the surrounding line uses template-like wording (e.g., inline 详见 docs/knowledge/xxx.md reference)
  3. Mark real references as:
    • Valid: target file exists
    • Stale: target file doesn't exist (action needed)

Phase 3: CLAUDE.md Compliance

Check that CLAUDE.md follows the architecture principle — it should describe the progressive disclosure system, NOT duplicate the detailed index.

  1. Count direct knowledge references: grep CLAUDE.md for docs/knowledge/ patterns
  2. Check architecture description: look for keywords indicating the architecture is described (e.g., "progressive disclosure", "progressive disclosure", "rules", "see details")
  3. Flag if CLAUDE.md contains more than 3 direct knowledge file references (suggests index leakage — the detailed index belongs in rules, not CLAUDE.md)

Exception: a summary table mapping rule files to knowledge domains is acceptable (it's architecture description, not a detailed index).

Phase 4: Report

Output a structured report:

==========================================
  CONTEXT AUDIT REPORT
  <project-name> | YYYY-MM-DD
==========================================

## Coverage Summary

Knowledge files: N total
  - M directly covered (cited from a rule)
  - K indirectly covered (cited from a rule-linked knowledge index)
  - X orphaned
  - Y deprecated
Rule files: N total
Stale references: N (placeholders excluded)

## Orphaned Knowledge Files (not indexed from any rule)

| File | Suggested Rule | Reason |
|------|---------------|--------|
| docs/knowledge/foo.md | rl-training.md | topic overlaps with RL training |

## Stale References (pointing to non-existent files)

| Rule File | Line | Reference | Status |
|-----------|------|-----------|--------|
| .claude/rules/bar.md | 42 | docs/knowledge/removed.md | file not found |

## CLAUDE.md Compliance

- Architecture description: [FOUND / MISSING]
- Direct knowledge refs: N (threshold: <=3)
- Status: [PASS / WARN — index leakage detected]

## Coverage Map

| Rule File | Knowledge Files Referenced |
|-----------|--------------------------|
| rl-training.md | reward, config, verl-integration, ... |
| cluster-ops.md | container-unify, mutagen-sync, ... |

## Deprecated Files

| File | Deprecation Notice |
|------|--------------------|
| docs/knowledge/experiments.md | migrated to experiment-registry |

==========================================

Phase 5: Fix Suggestions (optional)

If orphaned files are found, suggest concrete edits:

  • Identify the most relevant rule file based on topic overlap
  • Suggest the specific section where a > 详见 reference should be added
  • Provide the exact line to add (copy-pasteable)

Do NOT auto-apply fixes. Present suggestions and let the user decide.

Phase 6: Rule Slimming Methodology (when rules are bloated)

If audit reveals rules are too large (always-loaded context bloat), recommend the three-tier split approach (validated on agentic_umm 2026-04-25, reduced 8 rules / ~830 lines → 6 rules / 469 lines, -43%):

Tier 1 — Whole-file migration (highest ROI)

Identify rules that are pure how-to / reference manuals with no hard constraints. Move the entire file to docs/knowledge/, update the index pointer in CLAUDE.md/AGENTS.md to point at knowledge instead.

Signals: rule reads like a tutorial, no "must / must not / hard constraint" language, pure command recipes or organization conventions.

Tier 2 — In-file split (medium ROI)

For mixed rules, split into:

  • Keep in rule: hard constraints (project-specific invariants where violation causes incidents — data corruption, eval mismatch, lost work)
  • Move to knowledge: how-to procedures, detailed diagnostic stories, reference tables, command templates

Create a sibling knowledge file (e.g., <rule-name>-howto.md, <rule-name>-details.md) and replace the moved sections with a one-line > 详见 docs/knowledge/<...>.md.

Tier 3 — Index extraction (low-medium ROI)

If a rule contains a long flat table indexing many knowledge files (e.g., 20+ rows like "data/trajectory/evaluation"), extract that table into a dedicated docs/knowledge/<topic>-index.md. The rule keeps a single sentence pointing to the index. The index file is itself rule-referenced, so leaves remain reachable (legal two-tier pattern recognized in Phase 2).

Decision rules:

  • Hard constraint → rule (always-loaded)
  • How-to / reference / diagnostic detail → knowledge (on-demand)
  • Long flat index of knowledge files → dedicated index knowledge file
  • Cross-tool agents (e.g., Codex) that don't load .claude/rules/ → may need direct knowledge refs in AGENTS.md; this is acceptable cross-tool-compat redundancy, not a violation

Phase 7: Change Impact Matrix (forward audit, opt-in)

Phases 1-6 are reverse audits — start from existing files, check structural compliance. Phase 7 is the forward audit — start from recent changes, check whether the knowledge they introduced actually landed in every place it should.

This phase is opt-in: trigger it when the user supplies a change summary, or when the audit follows a sizable session (feature shipped, refactor done, bug fixed). Skip silently if there's nothing to forward-audit.

Inputs

Collect "new facts" introduced by recent work. Sources, in order of preference:

  1. User-supplied change summary (most reliable)
  2. git log --since="<session start>" --oneline + git diff of touched files
  3. Recent conversation context (least reliable — confirm with user before acting on it)

A "new fact" is anything that, if the next session reads stale docs, would mislead the agent. Examples: dependency swap (SQLite → PostgreSQL), new env var, renamed CLI flag, deprecated module, new invariant ("never call X directly"), new directory convention.

Mapping rules — fact type → expected locations

Fact type Should appear in
Tech stack swap (DB, framework, runtime) CLAUDE.md tech section + relevant rule + setup docs/README
New env var / config key runbook + CLAUDE.md (if always-needed) + .env.example
New hard constraint ("never X") a rule (always-loaded), not knowledge
New how-to / command recipe knowledge file + index reference from rule
Deprecated feature/file [DEPRECATED] marker + rule reference removed + replacement noted
Renamed module/function grep-replace across all docs + CLAUDE.md if mentioned
New directory or file convention a rule + CLAUDE.md repository structure section

Procedure

For each new fact:

  1. Determine fact type from the table above
  2. Enumerate expected locations
  3. For each expected location, grep for evidence the fact landed:
    • ✅ Updated: location contains the new fact
    • ❌ Missing: location exists but still describes the old state, or the fact is absent entirely
    • ⚠️ Conflict: location partially updated — old and new state coexist (often worse than fully stale, since it confuses the reader)
  4. Cross-project check: if this repo is depended on by another (monorepo sibling, paired client/server, etc.), flag whether the dependent's docs may need the same update

Output

Append to the Phase 4 report:

## Change Impact Matrix

Source: [user-supplied | git diff | conversation]
Window: <commit range or session start>

### Fact 1: <one-line description, e.g. "DB switched from SQLite to PostgreSQL">

| Expected location | Status | Evidence |
|-------------------|--------|----------|
| CLAUDE.md "Tech Stack" | ❌ Missing | line 42 still reads "SQLite" |
| docs/knowledge/db-schema.md | ⚠️ Conflict | lines 10-30 use Postgres, line 88 still has `sqlite3` import |
| .claude/rules/data-access.md | ✅ Updated | — |
| README.md install steps | ❌ Missing | step 3 still says "create sqlite db" |

Cross-project: agentic_umm-cli depends on this repo's DB schema → flag for review

### Fact 2: ...

Constraints

  • Phase 7 is still read-only — emit the matrix, don't apply edits
  • If no facts are supplied and no recent commits exist, skip silently rather than fabricate
  • Conflicts (⚠️) are higher priority than misses (❌) — call them out first

Constraints

  • Read-only — never modify files. This is an audit tool, not an auto-fixer.
  • Project-agnostic — works with any project that has docs/knowledge/ and .claude/rules/. Does not assume specific file names.
  • Follow the user's language — respond in whichever language the user uses.
  • Deprecated files are not failures — files with [DEPRECATED] markers are expected to exist without rule references. Report them for visibility but don't flag as errors.
  • Threshold, not perfection — the CLAUDE.md compliance check uses a threshold (<=3 direct refs). A summary table mapping rules to knowledge domains is architecture description, not index leakage.