quality-assurance-auditor

Quality Assurance Auditor

Global process collaboration constraints (drift prevention in long conversations)

• This skill must not be used as an isolated entry point. When the user requests a complete paper, Word generation, continuation of the process, or when the stage is uncertain, first return to paper-workflow-orchestrator to determine the current S0-S8 stage.
• Before starting or continuing a formal task of this skill, you must run:

  python .claude/skills/paper-workflow-orchestrator/scripts/workflow_guard.py --skill quality-assurance-auditor
  

• If the output is [WORKFLOW FAIL] or the report shows status != "PASS", stop this skill, and backfill the prerequisite stages according to the failure items in paper_output/qa/workflow_guard_report.json ; do not continue from memory.
• This skill only writes paper_output/ artifacts within the scope of its own contract; after completion, you must return to paper-workflow-orchestrator to determine the next step, and use context-memory-keeper to record completed artifacts, blockers, and the next step.
• In a long conversation, if the context grows long, the stage is uncertain, or the user invokes skills separately, first run:

  python .claude/skills/paper-workflow-orchestrator/scripts/workflow_guard.py --status
  

  Then read paper_output/qa/workflow_guard_report.json、paper_output/preflight_report.json、paper_output/input_manifest.json、paper_output/results/run_manifest.json and the upstream JSON contract of this skill, and continue according to the recommended_skill and next_action in the report.
• Before continuing the process, you must treat paper_output/context/workflow_memory.json as a long-term checkpoint record; if its current_step、next_step、recommended_skill and workflow_guard.py --status are inconsistent, the guard report shall prevail.
• After completing the artifacts of this skill each time, first return to paper-workflow-orchestrator or run workflow_guard.py --status, then update workflow memory:

  python .claude/skills/context-memory-keeper/scripts/update_workflow_memory.py
  

  After updating, read paper_output/context/workflow_memory.json / .mdto confirm that the next step and recommended skill have been recorded.

Execution contract

• Upstream inputs: preferentially read paper_output/plan/model_route.json、rubric_alignment.json、data_plan.json、visualization_plan.json、paper_output/figure_index.json、paper_output/results/model_results.json、paper_output/results/run_manifest.json、paper_output/results/metrics.json、paper_output/results/conclusions.json and paper_output/tables/table_index.json; if missing, fall back to paper_output/step1/problem_analysis.json。
• Before the formal draft, you must output paper_output/qa/evidence_gate_report.json and .md; the machine-readable report must record the SHA-256 of this audit's input.paper_output/tasks.json is only an optional checklist for legacy/quickstart and is not part of the formal S7 contract.
• Downstream handoff: after the official evidence gate PASSES, proceed directly to paper-formal-writer 's adaptive section writing.paper-micro-unit-generator Only when the S7 repair queue explicitly requires micro-repair does it participate in the formal process.
• Recommended next step: return to paper-workflow-orchestrator, and let the guard route to paper-formal-writer; the legacy/quickstart merged draft must not be treated as an officially audited passed draft.
• Failure fallback: if problem_files/ is empty, it should block; if the model route is missing, use problem analysis to generate the task; if problem analysis is also missing, only then use the general task template; if the formal evidence gate lacks run_manifest.json or the run records do not match, you must fall back to model-code-and-result-generator and rerun the modeling code.

Goal

• Insert "mandatory acceptance points" at every key node of paper generation, so that the next step can be taken only after the audit passes, preventing hidden laziness such as "meeting the word count but with logical errors" or "model substitution".
• Provide quantifiable pass/fail determinations and give a specific list of revisions, ensuring the final paper is both "thick" and "correct".
• Clearly distinguish between the "quickstart verification draft" and the "formal competition draft": the script running successfully does not mean the paper is qualified; when the result evidence is still only a skeleton, the final draft must not be delivered.
• Clearly define the boundaries of responsibility:evidence_gate.py Determine whether the results come from a reproducible run, whether the inputs and artifacts are still fresh, and whether the metrics and chart/table outputs are usable;paper-formal-writer/scripts/check_paper_format.py Assess the dynamic length, three-level headings, native formulas, in-text citations, figure and table references, and Word rendering quality. Only after both gates pass can it be called a formal draft.

When to Use

• After any skill step (problem parsing, model selection, structure design, micro-unit generation) is completed, the user wants to confirm, "Was this step really done correctly?"
• Before merging the full text, conduct a final consistency scan to prevent contradictions, missed constraints, and broken figure/table links.
• When the user suspects that "the AI secretly simplified the model" or that "the generated content is empty," immediately invoke this skill to perform a mandatory audit.

Inputs

• Required: the artifact to be audited (such as the model roadmap output by problem-doc-model-selector , the paper structure checklist, or a single micro-unit text).
• Required: the original competition problem PDF/Word content or excerpts of key constraints, for comparison.
• Optional: a user-defined list of scoring points (such as the official CUMCM rubric), which can override the default rules.

Outputs

• Audit report (JSON + human-readable):
  • checkpoint_id: the ID of this audit node
  • status：PASS / FAIL
  • score: a quantified score from 0–100
  • failures[]: a list of failed items, including "problem description," "location," and "suggested revision"
  • warnings[]: items that are acceptable but recommended for optimization
• If the status is FAIL, thenrefuse to proceed to the next step, and provide a "must-fix list"
• If the status is PASS, then provide a "verified passed" stamp and it is safe to continue

Audit Dimensions (can be toggled layer by layer)

1. Problem coverage
   • Whether all subproblems (Q1/Q2/Q3…) are 100% identified
   • Whether all explicit constraints (inequalities, boundaries, special conditions) are 100% covered
2. Model-task fit
   • Whether the model type (prediction/optimization/classification) is consistent with the task objective
   • Whether it has the capability to handle the given data format (time series → time-series model, cross-section → regression, etc.)
3. Logical consistency
   • Whether assumptions ⇄ model derivation ⇄ interpretation of results form a closed loop
   • Whether the symbol table contains conflicts or dual conventions (the same symbol with two meanings)
4. Scoring point alignment
   • Whether each high-scoring point (sensitivity analysis, controlled experiments, error quantification) is promised in a "locatable" way in the structure or text
5. Content completeness and anti-padding (High Priority)
   • No word-count padding: copying a paragraph and only replacing numbers, sequence numbers, a few synonyms, or connectives is all regarded as mechanical expansion and directly FAILs.
   • Reject repetition and placeholders: consecutive repetition, near-duplication after number normalization, template placeholders, and traces of "results to be filled in" all directly FAIL.
   • Structural check: check against the formal outline whether the core chapters and the evidence for each question are complete; empty background, repeated conclusions, or stacked figure/table descriptions cannot replace model derivation.
6. Evidence and run freshness
   • execution_provenance、run_manifest.json, the modeling script, input files, and output artifacts must match one another, and the recorded SHA-256 must equal that of the current file.
   • Computed metrics must be non-empty and finite;None, empty string,NaN、Inf or missing result_summary all directly FAIL.
   • If a chart or table is missing, an empty file, failed to generate, marked as a placeholder, or contains failure/placeholder descriptions, it must not be used as formal evidence.
7. Chart-body chain detection
   • Whether the body text says "see Figure 3" while Figure 3 actually exists and is numbered correctly
   • Whether the chart/table title and the body text description are semantically consistent
8. Formal delivery check
   • Word export, native formulas, body citations, three-level headings, and rendering quality are handled by paper-formal-writer ; this skill does not assemble the formal Word directly from final_paper.md .
   • Final acceptance must read format_check_report.json, confirm the status is PASS, the input hash is still fresh, and render_qa.status=PASS。

Workflow (can be nested into any stage)

1. Receive the artifact to be audited and the original competition problem
2. Scan item by item according to the dimensions specified by the user
3. Generate a PASS/FAIL report and a revision checklist
4. If FAIL, thenblock subsequent skill calls, forcing a return for revision; if PASS, then stamp and release
5. Optional: write the audit results to .audit_log.jsonlfor subsequent traceability

Usage example

• Input: (1) the "model route" output by problem-doc-model-selector, (2) the original competition problem text
• Behavior:
  • Check whether any subproblem is omitted → find that Q3 is not mentioned → FAIL
  • Report:failures: [{"issue": "子问题 Q3 未被识别", "suggestion": "在模型路线中补充 Q3 的任务拆解与指标定义"}]
  • The user must first revise the model route, and only after re-auditing passes can they proceed to structural design

Anti-laziness mechanism

• This skill never generates any new content, and only performs "Review & Reject"
• All judgment rules are open and transparent; users can customize thresholds or toggle dimensions
• On FAIL,force blocking, which cannot be bypassed through prompts and requires substantive revision

Notes

• Audit rules are aligned by default with the CUMCM national competition scoring standards, and can be overridden via the rubric= parameter
• Supports Chinese and English competition problems and outputs; the keyword library can be extended
• The audit itself does not depend on external large models; it is entirely based on rules and a keyword library, ensuring speed and impartiality

---

Appendix A: Script entry points (recommended)

The executable scripts of this skill are all placed under scripts/ .

Currently, scripts/pipeline.py is the legacy/quickstart basic check script, responsible for initializing directories, checking problem_files/ and generating as needed paper_output/tasks.json. Formal S6 uses evidence_gate.py; formal S7 does not depend on tasks.json。

scripts/evidence_gate.py is the evidence gate script before formal finalization, responsible for checking whether each question_id has real model results, finite and non-empty evaluation metrics, readable charts or tables, conclusion callback, and task tracking. official mode will recheck execution_provenance and paper_output/results/run_manifest.json, compare the current size and SHA-256 of modeling scripts, input files, and output artifacts, and reject code or data modified after running; missing result_summary, failed/empty/placeholder charts and tables, and no matching run records will also fail. It outputs paper_output/qa/evidence_gate_report.json and .md, and in the JSON's input_hashes records this gate input for subsequent formatting and workflow guard checks on whether the report is stale. If official mode does not pass, it returns a non-zero exit code; in quickstart mode it only gives a warning.

paper-formal-writer/scripts/check_paper_format.py is the formatting gate script after the formal draft is completed. It reads the dynamic length targets generated in the outline according to the number of sub-questions, and checks 1 / 1.1 / 1.1.1 third-level headings, the modeling/algorithm/results/verification for each question, figure and table references, closure between in-text citations and references, editable Word OMML formulas, duplicate paragraphs, and internal engineering jargon. The final deliverable must use --render required, convert the DOCX to PDF via LibreOffice and verify the page count and extractable text; the report also records the hashes of the source draft, DOCX, outline, index, and evidence report. It does not replace evidence_gate.py, but rather, after the evidence gate passes, prevents a Word document with unqualified content or layout from being called the final draft.

• If running legacy/quickstart and paper_output/plan/model_route.jsonexists, the pipeline will preferentially dynamically generate the micro-unit checklist according to the model route, scoring-point evidence, main model, verification plan, and recommended figures.
• If paper_output/plan/data_plan.json、visualization_plan.json and paper_output/figure_index.jsonexist, the script performs a lightweight evidence-chain check: it confirms that figure and table IDs, output paths, and data paths are traceable, but it will not block the entire process just because the planned figures have not actually been generated yet.
• If paper_output/results/model_results.json、metrics.json、conclusions.json and paper_output/tables/table_index.jsonexist, the script writes result_summary、key_metrics、tables、conclusions、evidence_status into each sub-question task for direct use by the micro-unit generator.
• If the model route contract does not exist but paper_output/step1/problem_analysis.jsonexists, the script dynamically generates the micro-unit checklist according to the real sub-questions, task types, recommended models, verification plans, and recommended figures.
• Only if no structured problem analysis exists does the script fall back to the generic task checklist template.
• If paper_output/tasks.json already exists, it is not overwritten by default; if you need to regenerate it according to the latest problem statement, set MATHMODEL_REGENERATE_TASKS=1 and then run it.

More detailed audit rules are written in this SKILL.md . When actually accepting a paper, the Agent must execute these rules in combination with the main text, problem statement, task checklist, and figure and table references, and must not treat merely getting the script to run as passing quality.

Run in the project root directory：

python .claude/skills/quality-assurance-auditor/scripts/pipeline.py

Run the evidence gate before the formal draft：

python .claude/skills/quality-assurance-auditor/scripts/evidence_gate.py

In quickstart verification, only look at warnings：

python .claude/skills/quality-assurance-auditor/scripts/evidence_gate.py --mode quickstart

Behavior: official mode reads the model route, real run ledger, results, metrics, conclusions, figures, and tables, and outputs a formal evidence gate with input hashes; pipeline only generates an optional task checklist for legacy/quickstart.

Directory conventions (aligned with the project globally)

• This skill will enforce that problem_files/ is non-empty.
• This skill uniformly produces formal gate reports under paper_output/qa/ ; legacy/quickstart checklists are still located at paper_output/tasks.json。

Before-and-after connection

• It often serves as the S6 global gate: it must pass before formal writing.
• After official PASS, return to paper-workflow-orchestrator, and usually enter paper-formal-writer; micro-units only handle local repairs explicitly queued by S7.

Constraints (must be followed)

• Memory Interaction (required):
  • After the audit passes: must call context-memory-keeper, recording the evidence gate status, report hash, and next stage.
• This skill is the S6 global gate: formal writing depends on a fresh PASS, not on the task checklist.
• The formal paper-micro-unit-generator entry must be triggered by repair_queue.json of micro-repair item triggered; only legacy/quickstart depends on tasks.json。
• If evidence_gate.py has not passed, it is forbidden to call final_paper.docx the final draft; you must return to model-code-and-result-generator or the code dedicated to the current competition problem, and fill in the real results, metrics, charts, tables, and conclusions.
• If paper-formal-writer/scripts/check_paper_format.py --render required has not passed, it is forbidden to call final_paper.docx the final draft; you must return to final_paper_source.md or the formatting stage to fix dynamic length, heading structure, chart explanations, native formulas, in-text citations, duplicate content, references, appendices, or rendering issues.
• When the user has already generated paper_output/final_paper_source.md , use S7 final audit and S8 format gate for final consistency checks.