research-guardian

Research Guardian — AI Research Quality Guardian System

Design Philosophy

The most dangerous thing about AI doing research is not "being unable to do it," but "doing it wrong while looking very right." The purpose of this skill is not to limit the AI's research capabilities, but to add verification gates at every key node, ensuring the output can withstand peer-review-level scrutiny.

Core Principle:Every research assertion must pass at least two independent verifications before being output.

Architectural Iron Rule:The agent that generates content can never evaluate its own output. The evaluation steps of all Gates must be executed by an independent subagent in a clean context. This is not an optimization suggestion — this is a hard requirement to prevent context rot and self-evaluation bias. See references/subagent-evaluation.md。

Performance Principle:Not everything needs to be fully enabled every time. First read references/quick-modes.md Select the mode (🟢 QUICK / 🟡 STANDARD / 🔴 FULL) and scenario-based Preset (ML submission / clinical report / social science survey / literature review / hypothesis generation). Use 10% of compute to catch 80% of problems, and automatically escalate when needed.

---

Trigger Conditions

When an AI agent is performing any of the following tasks, the corresponding protection module is automatically activated:

Task Type	Activated Module	Reference Document
Generate Hypothesis	Hypothesis Guard	references/hypothesis-validation.md
Literature Review	Literature Guard	references/literature-verification.md
Experimental Design	Experiment Guard	references/experiment-checklist.md
Results Analysis	Result Guard	references/result-integrity.md
Write Paper	Writing Guard	references/writing-verification.md
After any gate runs	Performance Metrics	references/performance-metrics.md
All gates (architecture layer)	Subagent Evaluation	references/subagent-evaluation.md
All gates (cross-cutting layer)	Logic Fallacy Chain	references/logic-fallacy-chain.md
Pipeline entry (read first)	Quick Modes	references/quick-modes.md
Parallel Execution Architecture	Parallel Execution	references/parallel-execution.md
All gate output formats	Issue Schema + Evidence	references/issue-schema.md
Frontmost of Pipeline	Paper Parser + Metadata	references/ingestion/paper-parser.md
After gate results are aggregated	Critique Synthesis	references/critique-synthesis.md
After gate results are aggregated	Red Flag Synthesis	references/red-flag-synthesis.md
Multiple papers	Batch Review Mode	references/modes/batch-review.md
Fabrication Detection	Audit Mode	references/modes/audit-mode.md
Paper Comparison	Compare Papers Mode	references/modes/compare-papers.md
Simulated Reviewer	Reviewer Personas	references/surpass/reviewer-personas.md
Predict Review Comments	Counter-Arguments	references/surpass/counter-arguments.md
Version Comparison	Version Diff	references/surpass/version-diff.md
Submission Risk Assessment	Rejection Risk Score	references/surpass/rejection-risk.md
Revision Suggestions	Fix Examples	references/surpass/fix-examples.md
Extension System	Extensibility Guide	references/extensibility-guide.md

Each module can run independently or be chained into a complete pipeline. The Performance Metrics module automatically collects performance data after each gate run, without additional triggering.

v1.7 Architecture Upgrade (referencing OpenReviewer design)：

1. Parallel Execution: After Pre-scan, all gates + logic chain are dispatched simultaneously and run in parallel. FULL mode drops from ~22 min to ~6.5 min (3.4x speedup). See references/parallel-execution.md。

2. Standardized Issue Schema: All gates output a unified JSON format, including evidence anchoring (precisely pointing to the original text location), confidence scores, and suggested_fix. See references/issue-schema.md。

3. Issue Normalization: Aggregator automatically deduplicates, calibrates severity, and ranks. The same issue independently detected by multiple gates → severity is automatically upgraded.

4. Cross-Section Consistency: 10 pairs of sections are forced to cross-validate (Abstract↔Results, Methods↔Results, etc.), executed at the Aggregator stage.

5. Citation Ethics: self-citation rate analysis, citation clustering detection, predatory journal detection. Executed synchronously in Literature Guard.

6. Anti-Hallucination Grounding: The system prompt of all subagents is forced to include anti-hallucination rules — a judgment without evidence cannot produce an issue.

Cross-cutting Module Description: Logic Fallacy Chain is not an independent 6th Gate, but a reasoning verification layer that runs through all Gates. Each Gate checks whether "the material is right"; Logic Chain checks whether "the reasoning holds". The two run simultaneously, each executed by an independent subagent. Automatic trigger correspondences:

• Hypothesis Guard → Logic Layer 1 (formal logic) + Layer 2 (causal reasoning)
• Literature Guard → Logic Layer 4 (argument structure)
• Experiment Guard → Logic Layer 2 (causal reasoning) + Layer 3 (statistical reasoning)
• Result Guard → Logic Layer 2 + 3 + 5 (causal + statistical + research-specific)
• Writing Guard → Logic Layer 1-5 (all)

---

Guardrail Process Overview

⚠️ Architecture Iron Rule: Each GATE below must be executed by an independent subagent, and must not be self-evaluated by the main agent that produced the research content. The subagent only receives "the output to be evaluated + the checklist", and does not receive the main agent's reasoning process or conversation history. See references/subagent-evaluation.md。

研究輸入（由 Research Agent 產生）
  │
  ▼
══════════════════════════════════════════════════════════
  Pre-scan (Gate 0) — 30 秒快速掃描 → 決定模式和 Preset
══════════════════════════════════════════════════════════
  │
  ▼ 平行派發（所有 gate 同時啟動，各自獨立 subagent）
  │
  ┌──────────┬──────────┬──────────┬──────────┬──────────┐
  ▼          ▼          ▼          ▼          ▼          ▼
┌────┐    ┌────┐    ┌────┐    ┌────┐    ┌────┐    ┌─────┐
│ G1 │    │ G2 │    │ G3 │    │ G4 │    │ G5 │    │Logic│
│假設│    │文獻│    │實驗│    │結果│    │寫作│    │Chain│
│    │    │+倫理│   │    │    │    │    │    │    │L1-5 │
└─┬──┘    └─┬──┘    └─┬──┘    └─┬──┘    └─┬──┘    └──┬──┘
  │         │         │         │         │           │
  └────┬────┴────┬────┴────┬────┴────┬────┴────┬──────┘
       │         │         │         │         │
       ▼         ▼         ▼         ▼         ▼
  ┌─────────────────────────────────────────────────┐
  │  Issue Normalization（去重 + 嚴重度校準）         │
  │  → 標準化 JSON Issue Schema                     │
  │  → 同一問題被 2+ gate 偵測 → severity 上調      │
  └──────────────────────┬──────────────────────────┘
                         ▼
  ┌─────────────────────────────────────────────────┐
  │  Cross-Section Consistency（10 對交叉驗證）       │
  │  Abstract↔Results, Methods↔Results,             │
  │  Hypothesis↔Conclusion, Figures↔Text ...        │
  └──────────────────────┬──────────────────────────┘
                         ▼
  ┌─────────────────────────────────────────────────┐
  │  Aggregator + Performance Metrics               │
  │  → 最終信心等級 + 效能摘要                       │
  │  → Anti-Hallucination check（無證據=移除 issue）  │
  └──────────────────────┬──────────────────────────┘
                         ▼
══════════════════════════════════════════════════════════
  結果回傳（只傳評估結論，不傳推理過程）
══════════════════════════════════════════════════════════
                         │
                         ▼
            ✅ 研究輸出（標準化 Issue 報告 + 信心等級）

---

Confidence Level Marking System

All research outputs that pass the guardrail process must carry a confidence level:

• 🟢 HIGH — passes all gates, has multi-source cross-validation, and all citations are confirmed to exist
• 🟡 MEDIUM — passes most gates, but some assertions have only a single source or cannot be fully verified
• 🔴 LOW — some gate was not passed or was skipped; the output is for reference only and requires manual review
• ⚫ UNVERIFIED — has not undergone any verification process; clearly marked as unverified AI-generated content

Each output paragraph should be marked with a confidence level, rather than marking it only once at the end of the document.

---

Global Default Rules (v1.1 — added based on validation testing)

Default Handling of Unreported Items

當研究輸出未報告某個 checklist 要求的資訊時：

未報告的項目類型           → 默認處理
────────────────────────────────────────
方法論關鍵資訊（隨機種子、power analysis、IRB）
                          → 默認 🔴 FAIL
統計報告細節（CI、效果量）  → 默認 🟡 WARN
可重現性資訊（code、env）  → 默認 🟡 WARN
搜索/驗證紀錄              → 默認 🔴 INCOMPLETE

原則：沉默不等於合規。如果 checklist 要求報告但輸出沒有報告，
     gate 不能假設「沒報告 = 沒問題」。

Pre-scan Step (Gate 0)

在正式進入 5 道 Gate 之前，先做一個 30 秒的快速掃描：

□ 整體語氣是否合理？
  - 通篇 "prove", "definitive", "revolutionary" → 立即紅旗
□ 結論的強度是否與摘要中描述的方法匹配？
  - 一個 dataset + 一個 baseline → "全球適用" → 明顯不匹配
□ 是否有明顯的結構缺失？
  - 無 Limitations → 紅旗
  - 無反面文獻 → 紅旗

Pre-scan 不取代任何 Gate，但可以在 10 秒內提供整體風險評估，
幫助 Orchestrator 決定要使用哪種 Subagent 策略（A/B/C）。

Cross-gate Contradiction Detection

Aggregator 在彙整報告時，必須檢查以下跨 gate 矛盾：

Gate 2 (Literature) 發現引用問題 → 自動觸發 Gate 4 (Result) 重檢
  理由：如果引用的基線數字是幻覺，比較結果也不可信

Gate 3 (Experiment) 發現設計缺陷 → 自動觸發 Gate 4 (Result) 重檢
  理由：設計有問題的實驗，結果的可信度也受影響

Gate 4 (Result) 發現 overclaim → 自動觸發 Gate 5 (Writing) 加嚴
  理由：結果段有 overclaim，Discussion 段很可能更嚴重

Logic Chain 發現因果謬誤 → 回饋到 Gate 1 (Hypothesis) 重檢
  理由：推理有問題可能意味著假設本身需要修正

---

Detailed Operation Guide for Each Gate

For detailed checklists and operating steps, see the respective reference documents:

• Hypothesis Validation: see references/hypothesis-validation.md

  • When to read: when the AI agent is generating new research hypotheses, research questions, or claiming to have discovered a new research direction

• Literature Validation: see references/literature-verification.md

  • When to read: when the AI agent is citing papers, conducting a literature review, or making arguments based on existing research

• Experiment Check: see references/experiment-checklist.md

  • When to read: when the AI agent is designing experiments, choosing methodologies, or calculating statistical parameters

• Result Validation: see references/result-integrity.md

  • When to read: when the AI agent is reporting numerical results, drawing charts, or drawing conclusions from data

• Writing Validation: see references/writing-verification.md

  • When to read: when the AI agent is writing a research report, paper paragraphs, or any research text to be publicly published

• Performance Measurement: See references/performance-metrics.md

  • When to read: When you need to evaluate the performance of the Guardian system itself, set up benchmark tests, build a performance dashboard, or detect system degradation
  • Also automatically referenced after every complete pipeline run, used to generate a performance summary

• Subagent Evaluation Specification: See references/subagent-evaluation.md

  • When to read: Required reading when integrating Guardian into an agent pipeline. Defines the context isolation strategy, subagent launch specifications, the list of content that must not be passed, the inter-rater reliability check method, and the evaluation strategy corresponding to the three risk levels
  • This is an architecture-level specification, and all gates must follow it

• Logical Fallacy Chain: See references/logic-fallacy-chain.md

  • When to read: Automatically launched in sync when each Gate runs. This is a cross-cutting module, not an independent Gate—it runs in parallel alongside each Gate, executed by an independent Logic Evaluator Subagent
  • Includes 5 layers of checks (formal logic, causal reasoning, statistical reasoning, argument structure, research-specific fallacies), and each Gate activates the corresponding layer
  • If compute is limited, at least activate Layer 2 (causal) + Layer 3 (statistical); these two layers cover the most common AI reasoning errors

• Quick Mode: See references/quick-modes.md

  • When to read:Read first every time Guardian starts. Determines which mode to run (🟢 QUICK / 🟡 STANDARD / 🔴 FULL) and which scenario-based Preset (ML submission / clinical report / social science survey / literature review / hypothesis generation)
  • Includes an automatic mode selector, progressive escalation rules, and performance metric comparison
  • Can save about 83% of compute, with interception rate dropping < 10%

• Parallel Execution Architecture: See references/parallel-execution.md

  • When to read: Required reading when integrating Guardian into an agent pipeline. After Pre-scan, all gates are dispatched in parallel, and FULL mode is 3.4x faster
  • Includes dependency analysis, dispatch specifications, API example code, and performance estimation

• Issue Schema + Evidence + Normalization + Citation Ethics + Anti-Hallucination: See references/issue-schema.md

  • When to read: The subagents of all gates must follow this format to output issues. The Aggregator uses this format for deduplication and severity calibration
  • Includes: standardized JSON issue format, evidence anchoring, 10 pairs of cross-section consistency checks, citation ethics checklist, anti-hallucination grounding rules

---

Known AI Research Failure Modes (Based on Literature)

The design of this skill is based on documented failure cases of AI research systems:

1. Hallucinated Citations: Generates nonexistent papers, including plausible authors, journals, and years
2. Novelty Misjudgment: Labels known concepts as "new discoveries" (e.g., treating micro-batching as an innovation)
3. Experimental Code Errors: It is known that 42% of automated experiments fail due to coding errors
4. Results Contradict Objectives: The experimental design is intended to optimize metric A, yet it reports improvement in metric B
5. Overgeneralization: Draws overly strong causal conclusions from limited data
6. Structural Defects: Missing figures or tables, duplicated paragraphs, placeholder text not removed
7. Outdated Citations: Heavy use of literature more than 5 years old, ignoring the latest advances
8. Score inflation: Systematically overestimating the quality of its own output in self-assessment
9. Context rot: In long conversations, the agent's judgment degrades as context expands
10. Self-evaluation bias: LLM evaluators systematically prefer LLM-generated content
11. Causal inference leaps: Drawing causal conclusions directly from observational data
12. Logical breaks silently pass through: Citations are correct and numbers are correct, but the reasoning logic is broken
13. Cross-section inconsistency: Numbers in the Abstract differ from the Results, analyses described in the Methods disappear from the Results
14. Citation ethics blind spots: Excessive self-citation, citation rings, selectively ignoring competing literature
15. Judgments without evidence: The AI evaluator itself may also hallucinate "problems"—judgments without source-text evidence should not be produced

Every gate in this skill is designed against at least one of the above failure modes. Items 8-10 are addressed by the subagent isolation architecture (see references/subagent-evaluation.md). Items 5, 11, and 12 are addressed by the Logic Fallacy Chain cross-cutting module (see references/logic-fallacy-chain.md). Item 13 is addressed by Cross-Section Consistency (see references/issue-schema.md). Item 14 is addressed by Citation Ethics (see references/issue-schema.md). Item 15 is addressed by Anti-Hallucination Grounding (see references/issue-schema.md）。

---

Integration Guide

As an intermediate layer of the Agent Pipeline

[Research Agent] → [Research Guardian Skill] → [Output]
                         │
                    每個 gate 可回傳：
                    - PASS：繼續
                    - FAIL + reason：退回修正
                    - WARN + detail：通過但標記

Integration with Existing Tools

• Web Search: Literature Guard will actively invoke search to verify whether citations exist
• Code Execution: Experiment Guard can execute statistical verification scripts
• File Creation: Writing Guard can output the final document with annotations

Pipeline Runner（scripts/runner.py）

A Python script manages the entire pipeline lifecycle:

python scripts/runner.py review paper.txt                 # Auto mode
python scripts/runner.py review paper.txt --mode full      # Full mode
python scripts/runner.py review paper.txt --preset ml      # ML preset
python scripts/runner.py resume                            # 從中斷處恢復
python scripts/runner.py status                            # 查看進度
python scripts/runner.py dashboard                         # 產生 HTML 儀表板

Checkpoint / Resume mechanism：

• Each time a gate is completed → automatically write to /tmp/research-guardian/checkpoints/
• If the pipeline is interrupted (crash, timeout, Ctrl+C) → the checkpoint is preserved
• resume command continues from the last checkpoint, and completed gates are not rerun
• Automatic escalation also has checkpoints (every step of QUICK → STANDARD → FULL is recorded)

Output location：

/tmp/research-guardian/
├── checkpoints/          ← 進度 checkpoint（JSON）
│   └── run-YYYYMMDD-HHMMSS-xxxx.checkpoint.json
└── output/
    ├── json/             ← 機器可讀報告
    ├── markdown/         ← 人類可讀報告
    ├── html/             ← HTML 報告
    └── dashboard.html    ← 互動式儀表板

JSON Schema（schemas/guardian.schema.json）

Defines all data structures: Issue, GateResult, Scorecard, RedFlag, RejectionRisk, PipelineCheckpoint, ReviewReport. The output of all gates must conform to this schema.

Minimal usable configuration

If only one gate can be enabled, prioritize enabling Literature Guard(citation verification), because hallucinated citations are currently the most common and most easily detected error in AI research.

Extension System

Need to add a new domain (such as physics, law), a new research design type, a new statistical framework, a new reviewer persona, or a new preset? See references/extensibility-guide.md. Each type of extension has a standard template, naming conventions, registration steps, and necessary regression testing requirements. Adding a domain sub-module only requires creating 1-3 files + registering in the parent document. Adding a Gate is a heavyweight change requiring modification of 8 files—in most cases, a sub-module or logic layer is sufficient.

---

Command Interface

Command	Description
/review	Review a single research output (automatically selects Quick Mode)
/review --mode full	Full review (all gates + logic chain)
/review --persona all	Simulate review using 5 reviewer personas
/review --persona statistician	Review with a specific persona
/review --diff v1 v2	Compare the differences between two versions
/batch-review	Batch review multiple studies
/audit	Strict fraud detection mode
/compare	Compare multiple papers and rank them
/risk	Calculate only the rejection risk score
/fix	Generate correction examples for the issues already found

---

Beyond features (what OpenReviewer does not have)

On top of covering all OpenReviewer features, this skill provides the following unique capabilities:

Reasoning layer

• Logic Fallacy Chain (5-layer reasoning verification): not only checking the material, but checking the reasoning itself
• Structural Overclaim detection: goes beyond signal word matching and uses 4 structural patterns to detect disguised overclaims

Quality assurance layer

• Subagent isolation + Anti Context Rot: prevents the evaluator's own judgment from degrading
• 30-case validation + 25 patches: every checklist item has been empirically tested
• Performance Metrics self-monitoring: the system tracks its own interception rate and precision

Performance layer

• Quick Modes（🟢🟡🔴）: tiered performance, not everything fully enabled every time
• Parallel execution: FULL mode 3.4x speedup
• Scenario-based Preset: automatic configuration for ML/clinical/social science/SR/proposal

Coverage layer

• Domain Sub-modules: biomedical, CS/ML, social science, and qualitative each have dedicated checklists
• Multilingual Overclaim: Chinese/Japanese/Korean/Spanish signal word detection
• Qualitative research support: saturation, coding rigor, and the four criteria for trustworthiness

Beyond layer (unique)

• Reviewer Persona Simulation: simulates 5 types of reviewers (statistician, domain expert, methodologist, skeptic, constructive mentor)
• Counter-Argument Generation: pre-generates reviewer challenges + suggested responses + preventive corrections
• Version Diff: analyzes differences between two versions of a paper and tracks the degree of improvement
• Rejection Risk Score: quantifies rejection risk + improvement priorities (ROI ranking)
• Fix Examples: not only says "change this here," but gives before/after example text for the correction