product-goal-based-audit

Product Goal-Based Audit

Systematic quality audit that extracts design expectations from project docs, confirms the project's actual commitments, configures expert audit roles, generates executable checks, scores the system, cross-validates findings, reviews the audit itself, and drives iterative improvement through audit-diagnose-fix-verify cycles.

Stance

Forensic auditor who trusts data over claims. Measures what the system actually does against what it was designed to do. Treats PASS as "not yet proven wrong" rather than "confirmed working." Does not accept surface-level scores: distribution, trends, causality, expert disagreement, and missing evidence matter more than aggregate PASS/FAIL.

Subcommands

Command	Phase(s)	Description
mine	1	Extract expectations from project docs
instrument	2	Generate audit scripts from expectations
audit	3	Run audit, produce scored report
deep-dive	4	Analyze distributions, trends, causality beyond PASS/FAIL
diagnose	5	Identify root causes, build causal chains
prescribe	6	Generate structured fix prompts with dependency ordering
verify	7	Re-audit after fixes, compare scores, check for regressions
full	1-7	Complete audit cycle with iteration

Default behavior: If no subcommand, check project state:

• No EXPECTATIONS.md exists → start with mine
• Has expectations but no audit script → start with instrument
• Has audit script → start with audit
• Has recent audit report → ask user which phase to enter

Audit directory: Unless the user provides another location, write audit artifacts under {project-path}/.product-audit/:

• .product-audit/COMMITMENT_REVIEW.md
• .product-audit/EXPECTATIONS.md
• .product-audit/audit.sh or equivalent generated checker
• .product-audit/AUDIT_SCRIPT_REVIEW.md
• .product-audit/AUDIT_RESULT_REVIEW.md
• .product-audit/reports/
• .product-audit/state.md for iteration history, hashes, active roles, and latest report paths

Commitment Confirmation And Audit Self-Review

Before finalizing expectations, mine commitment candidates into COMMITMENT_REVIEW.md, classify each claim, self-review the mining result, and ask the user to confirm the real current commitments. If the user requests a trial audit before confirmation, mark the scope as provisional in expectations, reports, and state.

Before trusting generated audit infrastructure, review the audit itself:

• Phase 1: commitment mining self-review
• Phase 2: AUDIT_SCRIPT_REVIEW.md
• Phase 3: AUDIT_RESULT_REVIEW.md

Audit outcomes must be labelled trusted, provisional, or blocked. Self-review is adversarial confidence building, not proof of perfect coverage.

Adversarial Benchmark Mode

Use adversarial benchmark cases when changing the audit workflow or validating audit-of-audit behavior. The benchmark suite lives under benchmarks/adversarial/ and must include cases for roadmap traps, marketing traps, hidden critical promises, conflicting docs, fake PASS scripts, CRITICAL downgrades, manual coverage inflation, cumulative metric traps, pretty but wrong reports, expert disagreement, document-only specs, and external dependency boundaries.

Expert Panel Mode

When the audit goal or artifact set is broad enough, configure an expert audit panel before generating expectations. Load references/expert-role-taxonomy.md to select roles, then load references/multi-agent-orchestration.md when the runtime can run subagents or when sequential independent passes are useful.

Typical roles include product/requirements, architecture, engineering, test strategy, security, reliability/operations, performance, data, UX/accessibility, delivery/DevOps, compliance/privacy, documentation/DX, and AI behavior.

Expert panel outputs are evidence, not decoration:

• Role coverage summary
• Finding matrix with role, evidence, severity, confidence, and cross-check status
• Duplicate review results for CRITICAL or ambiguous expectations
• Disagreement ledger with adjudication
• Integrated risk list carried into diagnosis and prescriptions

Red Lines

1. No vague expectations — Every expectation must have a quantifiable threshold. Reject "should work well."
2. No uncovered expectations — Automated checks must cover >= 95% of defined expectations. Manual/INFO stubs are documented gaps and do not count toward automated coverage.
3. No incomplete fix prompts — Each must have: current data, source paths, verification method, target value, dependencies.
4. No monitor tampering — Audit infrastructure (scripts, expectations, reports) is read-only during fix execution.
5. No skipped gates — Each phase transition requires its gate checklist to pass.
6. No silent CRITICAL downgrades — A CRITICAL severity cannot become INFO without an explicit Condition field.
7. No cumulative-only traps — Expectations for ongoing operations must use incremental metric type, not cumulative.
8. No unbounded iteration — Maximum 5 audit-fix rounds without explicit user override.
9. No unconfirmed commitment scope — Final expectations require user-confirmed commitments or visible provisional status.
10. No trusted audit without audit-of-audit — Script and report self-reviews must pass before diagnosis or prescriptions rely on a report.

Scoring Overview

• Weights: CRITICAL=3x, HIGH=2x, MEDIUM=1x, LOW=0.5x
• Results: PASS=full, WARN=half, FAIL=0, INFO=excluded
• CRITICAL blocker: Any CRITICAL FAIL → grade capped at C maximum
• Grades: A>=90, B>=75, C>=60, D>=40, F<40 (A/B require 0 CRITICAL FAILs)
• Full formula: references/scoring-formula.md

Acceptance Criteria

1. Generated EXPECTATIONS.md covers confirmed extractable commitments (coverage manifest produced)
2. Commitment confirmation is recorded, or the audit is visibly marked provisional
3. Audit script runs without errors on first invocation and produces a scored report
4. Script and result self-reviews pass before diagnosis or prescriptions trust the report
5. After 1-3 fix rounds, measurable score improvement with no undetected regressions
6. All 12 review findings from the original brief are addressed (see Finding Map below)

Phase Modules (Lazy-Loaded)

When entering a phase, Read the corresponding file before executing any phase logic.

Phase	File	Key Deliverable
1: Mine	phases/phase-1-mine.md	EXPECTATIONS.md + coverage manifest + audit panel
2: Instrument	phases/phase-2-instrument.md	Audit script + coverage verification + self-test
3: Audit	phases/phase-3-audit.md	Scored report + expert finding matrix
4: Deep Dive	phases/phase-4-deep-dive.md	"Surface PASS, Deep Concern" findings + cross-validation
5: Diagnose	phases/phase-5-diagnose.md	Causal graph + role-attributed root causes
6: Prescribe	phases/phase-6-prescribe.md	Structured fix prompts with dependency ordering and role evidence
7: Verify	phases/phase-7-verify.md	Comparison report + termination evaluation

References (On-Demand)

File	Load When
references/expectation-template.md	Phase 1: writing expectations
references/project-type-profiles.md	Phase 1: classifying project type
references/expert-role-taxonomy.md	Phase 1: selecting audit roles; Phase 3-4: role packets
references/multi-agent-orchestration.md	Phase 3-4: subagent or sequential independent audit passes
references/commitment-confirmation.md	Phase 1: claim classification, user confirmation, provisional scope
references/audit-of-audit.md	Phase 1-3: commitment, script, and report self-review
references/adversarial-self-audit-benchmarks.md	Workflow changes: benchmark case design and release blockers
references/scoring-formula.md	Phase 3: computing scores
references/deep-dive-framework.md	Phase 4: structuring deep analysis
references/fix-dependency-graph.md	Phase 6: ordering fixes
references/iteration-bounds.md	Phase 7: evaluating termination
references/report-templates.md	Phase 3, 7: formatting reports

Gate System

All 7 phase-transition gates are consolidated in gates/gate-checklist.md. Each gate is a binary checklist — all items must pass before proceeding.

Finding Map (Traceability)

Original Finding	Addressed In
P0-1: Coverage gap (26 defined, 16 implemented)	Phase 2 coverage gate
P0-2: Severity inconsistency (CRITICAL→INFO)	Expectation Condition field
P0-3: Cumulative metrics hide failures	Metric Type field
P1-4: Deep Dive can't generalize	Deep Dive Metadata per expectation
P1-5: Scoring ceiling (CRITICAL FAIL = B)	CRITICAL blocker rule
P1-6: No iteration bounds	5 termination rules
P2-7: Backend-only	5 project type profiles
P2-8: No fix dependency handling	depends_on / conflicts_with
P2-9: Audit script untested	Dry-run + lint gate
P3-10: No auto-generation from code	Code artifact scanning in Phase 1
P3-11: No historical trends	Iteration history + ASCII trends
P3-12: Monitor protection verbal only	Tamper hashes + hash verification