quality-assurance-auditor

Quality Assurance Auditor

Global process collaboration constraints (long-conversation drift prevention)

• This skill must not be used as an isolated entry point. When the user requests a complete paper, Word generation, process continuation, or the stage is uncertain, first return to paper-workflow-orchestrator to determine the current S0-S8 stage.
• Before starting or continuing the formal task of this skill, you must run:

  python .claude/skills/paper-workflow-orchestrator/scripts/workflow_guard.py --skill quality-assurance-auditor
  

• If the output is [WORKFLOW FAIL] or reports status != "PASS", stop this skill and backfill the preceding stages according to the failure items in paper_output/qa/workflow_guard_report.json ; do not continue from memory.
• This skill only writes the paper_output/ artifacts within its own contract scope; after completion, you must return to paper-workflow-orchestrator to determine the next step, and use context-memory-keeper to record completed artifacts, blockers, and the next step.
• In a long conversation, if the context grows long, the stage is uncertain, or the user invokes skills separately, first run:

  python .claude/skills/paper-workflow-orchestrator/scripts/workflow_guard.py --status
  

  Then read paper_output/qa/workflow_guard_report.json、paper_output/preflight_report.json、paper_output/input_manifest.json、paper_output/results/run_manifest.json and this skill's upstream JSON contract, and continue according to the recommended_skill and next_action in the report.
• Before continuing the process, you must treat paper_output/context/workflow_memory.json as a long-term breakpoint record; if its current_step、next_step、recommended_skill and workflow_guard.py --status are inconsistent, the guard report shall prevail.
• After completing this skill's artifacts each time, first return to paper-workflow-orchestrator or run workflow_guard.py --status, then update workflow memory:

  python .claude/skills/context-memory-keeper/scripts/update_workflow_memory.py
  

  After updating, read paper_output/context/workflow_memory.json / .mdto confirm that the next step and recommended skill have been recorded.

Execution contract

• Upstream inputs: preferentially read paper_output/plan/model_route.json、rubric_alignment.json、data_plan.json、visualization_plan.json、paper_output/figure_index.json、paper_output/results/model_results.json、paper_output/results/run_manifest.json、paper_output/results/metrics.json、paper_output/results/conclusions.json and paper_output/tables/table_index.json; if missing, fall back to paper_output/step1/problem_analysis.json。
• Must output:paper_output/tasks.json, and ensure that the paper_output/micro_units/ directory exists; before formal drafting, you must output paper_output/qa/evidence_gate_report.json and paper_output/qa/evidence_gate_report.md。
• Downstream handoff:paper-micro-unit-generator should only generate the main text after tasks.json exists; the task must retain the model route, verification plan, chart suggestions, scoring points, result summary, metrics, tables, and conclusion fields.
• Recommended next step: after passing before main-text generation, proceed to paper-micro-unit-generator; if final_paper.md or final_paper.docxhas already been generated, perform a final consistency check and return to paper-workflow-orchestrator for aggregation.
• Failure fallback: if problem_files/ is empty, it should block; if the model route is missing, use the problem analysis to generate the task; only if the problem analysis is also missing, use the generic task template; if the formal evidence gate lacks run_manifest.json or the run record does not match, you must fall back to model-code-and-result-generator to rerun the modeling code.

Goal

• Insert "mandatory acceptance points" at every key node of paper generation; only after the audit passes can you proceed to the next step, preventing hidden laziness such as "word count meets the standard but logic is wrong" or "model substitution".
• Provide quantifiable pass/fail determinations and give a specific revision checklist to ensure the final paper is both "thick" and "correct".
• Clearly distinguish between a "quickstart verification draft" and a "formal competition paper": a script running successfully does not mean the paper is qualified; when the result evidence is still a skeleton, the final draft must not be delivered.
• Clearly define responsibility boundaries:evidence_gate.py Judge only whether the actual results, metrics, charts, tables, and conclusion evidence are sufficient; formal paper structure, word count, three-level headings, figure/table citations, and Word formatting are handled by paper-formal-writer/scripts/check_paper_format.py judge. Only after both gates pass can it be called a formal draft.

When to Use

• After any skill step (problem statement analysis, model selection, structure design, micro-unit generation) is completed, the user wants to confirm, "Was this step really done correctly?"
• Before merging the full text, perform a final consistency scan to prevent contradictions, omitted constraints, and broken figure/table links.
• When the user suspects that "the AI secretly simplified the model" or that "the generated content is hollow," immediately invoke this skill for a mandatory audit.

Input

• Required: the artifact to be audited (such as problem-doc-model-selector the output model roadmap, paper structure checklist, or a single micro-unit text).
• Required: the original competition problem PDF/Word content or excerpts of key constraints, for comparison.
• Optional: a user-defined scoring-point checklist (such as the official CUMCM rubric), which can override the default rules.

Output

• Audit report (JSON + human-readable):
  • checkpoint_id: the ID of this audit node
  • status：PASS / FAIL
  • score: 0–100 quantitative score
  • failures[]: list of failed items, including "problem description," "location," and "suggested revision"
  • warnings[]: acceptable but recommended-for-optimization items
• If the status is FAIL, thenrefuse to proceed to the next step, and provide a "must-fix list"
• If the status is PASS, then provide a "verified passed" stamp, and it is safe to continue

Audit dimensions (can be toggled layer by layer)

1. Problem coverage
   • Whether 100% of all subproblems (Q1/Q2/Q3…) are identified
   • Whether 100% of all explicit constraints (inequalities, boundaries, special conditions) are covered
2. Model-task fit
   • Whether the model type (prediction/optimization/classification) is consistent with the task objective
   • Whether it has the capability to handle the given data format (time series → time-series model, cross-section → regression, etc.)
3. Logical consistency
   • Whether assumptions ⇄ model derivation ⇄ interpretation of results form a closed loop
   • Whether the symbol table has conflicts or dual meanings (the same symbol with two meanings)
4. Scoring-point alignment
   • Whether each high-scoring point (sensitivity analysis, controlled experiments, error quantification) is "locatably" promised in the structure or text
5. Mechanical repetition detection (High Priority)
   • Padding for word count is allowed: academic connectives such as "with the development of…" and "in summary" may be used to increase length,notdetermined as Fail.
   • Crack down on mechanical copying: strictly checkfully repeated consecutive paragraphs(such as the same passage appearing twice in a row),unreplaced placeholders(such as the appearance of the words "supplementary explanation" with repeated content). If this kind of "messy" repetition is found, directly FAIL.
   • Structure check: compare with paper_prompt_default.md, focusing on whether core chapters are omitted; specific paragraph formatting may be appropriately relaxed.

Additional artifacts

• Word export (docx): After the audit passes, if there exists final_paper.md, it must be converted to final_paper.docxusing a conversion tool (such as pandoc or python-docx) as the final deliverable. Do not deliver Markdown only.

1. Figure-text chain check
   • Whether the text contains "see Figure 3" while Figure 3 actually exists and is numbered correctly
   • Whether the figure/table caption and the text description are semantically consistent

Workflow (can be nested into any stage)

1. Receive the artifact to be audited and the original competition problem
2. Scan item by item according to the dimensions specified by the user
3. Generate a PASS/FAIL report and a modification checklist
4. If FAIL, thenblock subsequent skill calls, mandatorily returning for modification; if PASS, stamp and release
5. Optional: write the audit results to .audit_log.jsonl, for subsequent traceability

Usage example

• Input: (1) the "model route" output by problem-doc-model-selector, (2) the original competition problem text
• Behavior:
  • Check whether any sub-problem is missing → find that Q3 is not mentioned → FAIL
  • Report:failures: [{"issue": "子问题 Q3 未被识别", "suggestion": "在模型路线中补充 Q3 的任务拆解与指标定义"}]
  • The user must first correct the model route and pass the audit again before entering structural design

Anti-laziness mechanism

• This skill never generates any new content, and only does "Review & Reject"
• All judgment rules are open and transparent; users can customize thresholds or toggle dimensions
• On FAILforcefully block, cannot be bypassed via prompt, and substantive modification is required

Remarks

• Audit rules are aligned with the CUMCM national competition scoring standards by default, and can be overridden via the rubric= parameter
• Supports Chinese and English competition problems and outputs; the keyword library is extensible
• The audit itself does not depend on an external large model; it is entirely based on rules and a keyword library, ensuring speed and non-bias

---

Appendix A: Script entry points (recommended)

The executable scripts of this skill are all placed under scripts/ .

Currently scripts/pipeline.py is the basic gatekeeping script, responsible for initializing directories, checking problem_files/ and generating paper_output/tasks.json。

scripts/evidence_gate.py is the evidence gatekeeping script before formal drafting, responsible for checking whether each question_id has real model results, evaluation metrics, figure or table evidence, conclusion callback, and task tracking. In official mode it also checks model_results.json for each formal result's execution_provenance, confirming that source_code_path exists,run_command is non-empty,run_exit_code=0 and that the output artifact is traceable; it also reads paper_output/results/run_manifest.json, confirming that the results indeed come from the actual run records of the unified entry point run_modeling.py . Results without a real code execution source or without a matching run_manifest record must not pass. It outputs paper_output/qa/evidence_gate_report.json and paper_output/qa/evidence_gate_report.md. In official mode, failure returns a non-zero exit code; in quickstart mode it only gives a warning.

paper-formal-writer/scripts/check_paper_format.py is the format gatekeeping script after formal drafting, responsible for checking whether final_paper_source.md reaches the 18000-25000 target, whether there are 1 / 1.1 / 1.1.1 level-3 headings, whether each question has modeling/algorithm/results/verification, whether figures and tables are cited in the text, and whether references and appendices are complete. It does not replace evidence_gate.py, but after the evidence gate passes, it continues to prevent a Word document with low word count and low formatting quality from being called the final draft.

• If there exists paper_output/plan/model_route.json, the script will dynamically generate the micro-unit checklist based primarily on the model route, rubric evidence, main model, validation plan, and suggested figures.
• If paper_output/plan/data_plan.json、visualization_plan.json and paper_output/figure_index.jsonexist, the script will perform a lightweight evidence chain check: confirming that figure IDs, output paths, and data paths are traceable, but it will not block the entire workflow just because the planned figures have not yet actually been generated.
• If paper_output/results/model_results.json、metrics.json、conclusions.json and paper_output/tables/table_index.jsonexist, the script will write result_summary、key_metrics、tables、conclusions、evidence_status into each sub-question task for direct use by the micro-unit generator.
• If the model route contract does not exist but paper_output/step1/problem_analysis.jsondoes, the script will dynamically generate the micro-unit checklist based on the actual sub-questions, task types, recommended models, validation plan, and suggested figures.
• Only if the structured problem analysis does not exist will the script fall back to the generic task checklist template.
• If paper_output/tasks.json already exists, it will not be overwritten by default; if you need to regenerate it based on the latest problem statement, set MATHMODEL_REGENERATE_TASKS=1 and then run again.

More detailed audit rules are written in this SKILL.md . When actually reviewing the paper, the Agent must execute these rules in conjunction with the main text, problem statement, task checklist, and figure references, and must not treat merely getting the script to run as passing quality.

Run in the project root directory：

python .claude/skills/quality-assurance-auditor/scripts/pipeline.py

Run the evidence gate before formal drafting：

python .claude/skills/quality-assurance-auditor/scripts/evidence_gate.py

During quickstart validation, only look at warning：

python .claude/skills/quality-assurance-auditor/scripts/evidence_gate.py --mode quickstart

Behavior: initialize directories → check whether problem_files/ is empty (block if it fails) → preferentially read paper_output/plan/model_route.json and rubric_alignment.json → read the data/figure/result/table contracts for lightweight evidence chain hints → fall back to reading paper_output/step1/problem_analysis.json → generate dynamic paper_output/tasks.json → report the current micro-unit completion progress and scan for placeholder traces.

Directory conventions (aligned with the project globally)

• This skill will enforce that problem_files/ is non-empty.
• This skill uniformly produces the task checklist and micro-unit directory under paper_output/ .

Upstream and downstream connections

• It often serves as a global gate: it is recommended to run it once before "generating the main text/merging the full text".
• It is usually followed by:paper-micro-unit-generator(generating micro-units and merging) or returning to paper-workflow-orchestrator to continue the complete workflow.

Constraints (must be followed)

• Memory Interaction (required):
  • After the audit passes: you must call context-memory-keeper, and update the "audit passed status" and the "overview of the generated task checklist" to Short-term Workbench。
• This skill is a global gate: when the user wants to enter the "generate main text/merge full text/deliver paper" stage, the directory check and task checklist generation of this skill must be passed first.
• When paper_output/tasks.json has not been generated, it is forbidden to directly enter paper-micro-unit-generator。
• If evidence_gate.py has not passed, it is forbidden to call final_paper.docx the final draft; you must return to model-code-and-result-generator or the current competition-specific code to fill in the real results, metrics, figures, tables, and conclusions.
• If paper-formal-writer/scripts/check_paper_format.py has not passed, it is forbidden to call final_paper.docx the final draft; you must return to final_paper_source.md to fill in the main text length, heading structure, figure explanations, references, or appendix code descriptions.
• When the user has already generated paper_output/final_paper.md , it is recommended to call this skill again for a final consistency check to ensure that "every question has a conclusion, figures can be located, and citations are not broken".