The last line of defense before research deliverables are handed over. The error rate of AI-generated URLs is about 60%, and precise numbers often have "creative merging" hallucinations - this skill automatically executes the four core methods of SOP P4 (SIFT / CRAAP / four-dimensional triangulation / 7-item AI hallucination checklist), outputs a risk list and action recommendations, and a human makes the final call on how to revise the report.
Core boundaries
- Do: verify URL authenticity, number traceability, source evaluation, triangulation, contradiction detection, output a risk list
- Do not: automatically fix the report (only output recommendations, humans decide how to revise); redo research (that is deep-research's job)
Configuration parameters
Users can specify via natural language, or use default values:
| Parameter | Value | Default | Description |
|---|---|---|---|
mode |
light / deep | light | Light mode 5 minutes / deep mode 15-20 minutes (adds all 5 CRAAP dimensions + four-dimensional triangulation) |
target |
full / point | full | Full report check / single-point data check |
whitelist_boost |
on / off | on | Enable mandatory-source whitelist weighting (iResearch, QuestMobile, Juliang Suanshu, etc.) |
language |
cn / en / both | both | Source language preference |
Example triggers:
- "Check this research report" (full + light, default)
- "Deep-check this report" (full + deep)
- "Is this 9.4 million figure real" (point)
Execution flow (5 steps)
Step 1: Parse input
Determine the user input type:
-
Full report check(
target: full):- User provides a markdown file path → use
Readto read - User pastes report content → process directly
- User says "that report just now" → extract the most recent report text from the conversation history
- User provides a markdown file path → use
-
Single-point check(
target: point):- Extract the "data + source" combination from the statement
- For example: "China has 9.4 million programmers, from the Ministry of Industry and Information Technology" → data = 9.4 million / subject = Chinese programmers / source = Ministry of Industry and Information Technology
Output a structured checklist of verification tasks (URL list, number list, core source list) for parallel processing in subsequent steps.
Step 2: URL health check
This is the highest-priority step - AI-hallucinated URLs account for about 60%, so if you do not verify them, there will almost certainly be problems.
Method:
-
Use
GreporBashregex toextract all URLs from the report:- Markdown links:
\[.*?\]\((https?://[^)]+)\) - Plain-text URLs:
https?://[^\s\)\]"',。、;]+ - After deduplication, obtain the URL list
- Markdown links:
-
In parallelrun
WebFetch:- on each URL to verify the HTTP status code (200 = pass, 404/403/5xx = fail)
- Retrieve the page title and body summary
- Compare keyword overlap with the citation context (50 characters before and after)
-
Output three lists:
- ✅ Accessible and relevant: status 200, content overlap with context ≥3 keywords
- ⚠️ Accessible but content does not match: Status 200, but the content is unrelated to the context (the most covert AI hallucination)
- ❌ Invalid: 404 / 403 / 5xx / DNS resolution failure
Key constraints: When there are many URLs (>20), process them in parallel batches (5 per batch) to avoid sending too many WebFetch requests at once.
Step 3: Precise number traceability
Prevent the AI from "creatively merging" numbers that do not exist.
Approach:
-
Use
Grepto extract all "precise number" patterns from the report:- Amounts:
\$\d+(\.\d+)?[BMK]?/¥\d+(\.\d+)?[万亿]? - Percentages:
\d+(\.\d+)?% - Large numbers:
\d+(,\d{3})+or\d+\s*(万|亿|百万|千万|million|billion) - Growth rates:
(YoY|同比|环比|CAGR)\s*[+-]?\d+(\.\d+)?%
- Amounts:
-
For each number, findthe nearest cited source(the URL or institution name in the preceding or following paragraphs).
-
Use
WebSearchfor reverse lookup: use "number + subject + time" as the query to see whether the original source can be found.- For example:
"940 万" 中国程序员 工信部 2024
- For example:
-
Label each number:
- ✅ Original text source found: This precise number can be seen in the original source
- ⚠️ Fuzzy match only: Related data can be found, but it is not the exact same number (difference < 10%)
- ❌ Not found in original text: Cannot be found in the cited source, a strong indicator of AI hallucination risk
Step 4: CRAAP source evaluation (only the deep mode executes the complete 5 dimensions)
For eachcore source(sources cited ≥2 times), perform a 5-dimension evaluation.
Automatic scoring logic(see reference.md §1):
| Dimension | Scoring basis |
|---|---|
| Currency Timeliness | WebFetch to get the publication date → time since then <6 months = 5 / 6-12 months = 4 / 1-2 years = 3 / 2-3 years = 2 / >3 years = 1 |
| Relevance Relevance | Degree of keyword overlap between the source content and the cited argument (≥5 = 5 / 4 = 4 / 3 = 3 / 2 = 2 / 1 = 1) |
| Authority Authority | Domain lookup whitelist (iResearch / QuestMobile / Juliang Suanshu / 36Kr / listed company financial reports → 5 / mainstream media → 3-4 / unknown blogs → 1-2) |
| Accuracy Accuracy | Whether the argument is supported by ≥2 independent sources (≥3 sources = 5 / 2 sources = 4 / 1 source = 2) |
| Purpose Purpose | Whether it is commercial/marketing/PR (academic/official reports = 5 / neutral media = 4 / industry analysis = 3 / company PR = 2 / stakeholder statements = 1) |
Each core source receives a total score of 0-25.Red flag rules: Any dimension ≤2 → mark red (detailed in reference.md §1).
Light mode skips this step, performing only the most basic ≥2 source cross-check.
Step 5: Triangulation + contradiction detection + 7-item AI hallucination checklist + output report
5a. Triangulation (key arguments)
- Automatically identify "conclusive arguments" in the report:
- Bold paragraphs (
\*\*...\*\*) - containing keywords such as "research shows", "data shows", "survey finds"
- paragraphs with high number density
- Bold paragraphs (
- For each argument, count the sources supporting itNumber of independent sources(deduplicated by domain)
- Markers:
- ✅ ≥2 independent sources → triangulation verified
- ⚠️ 1 source → strong prompt to add a reverse source
- ❌ 0 sources → purely AI-generated, must be deleted or sources added
5b. Additional work for the deep tier: four-dimensional triangulation verification
See reference.md §2 for implementation details covering 4 categories (data sources / methodology / investigators / theory).
5c. AI hallucination 7-item checklist
Mark each item as ✅ / ⚠️ / ❌, see reference.md §3:
- Precise number traceability——whether each number can be traced back to its original source
- Misattribution check——whether data has been incorrectly attributed to other companies/periods/markets
- Source link verification——already completed in Step 2
- Logical consistency——whether the data/conclusions in different sections are self-consistent
- Over-extrapolation check——whether limited data has been over-generalized
- Timeline correctness——whether the order of events and causal chronology are reasonable
- Comprehensive hallucination check——whether the AI "creatively merged" data from multiple sources into conclusions that do not exist
5d. Contradiction detection
When the same metric appears multiple times in different sections of the report, compare the values. Difference >5% → mark as a contradiction point.
5e. Output verification report
Generate a markdown report in the format below, and output it directly (do not archive it unless the user explicitly requests it).
Output report format
# 📋 事实核查报告
> **模式**:{light | deep} **覆盖**:X 个 URL / Y 个数据点 / Z 个核心信源
> **核查时间**:YYYY-MM-DD HH:MM
## 总览
- ✅ 通过:X 项
- ⚠️ 待补强:X 项
- ❌ 不通过:X 项
- **综合可信度评分:X / 100**
## 一、URL 健康度(X / Y 个)
| URL(截断) | 状态 | 上下文匹配 | 风险 |
|---|---|---|---|
| https://... | ✅ 200 | ✅ 相关 | — |
| https://... | ❌ 404 | — | **必须删除引用** |
| https://... | ✅ 200 | ⚠️ 不匹配 | **疑似 AI 幻觉,必须核实** |
## 二、数据点溯源(X / Y 个)
| 数字 | 引用源 | 反查结果 | 风险等级 |
|---|---|---|---|
| 940 万程序员 | 工信部 2024 | ✅ 知乎/EET-China 多源确认 | 低 |
| $4.2B 市场规模 | 报告未注明 | ❌ 反查无结果 | **高(疑似幻觉)** |
## 三、CRAAP 评估(核心信源,仅 deep 模式)
| 信源 | C | R | A | A | P | 总分 | 红旗 |
|---|---|---|---|---|---|---|---|
| 艾瑞咨询 2025 报告 | 5 | 5 | 5 | 4 | 4 | 23 | — |
| 某不知名博客 | 3 | 4 | 1 | 1 | 2 | 11 | 🚩 权威性 / 准确性 |
## 四、三角验证缺口
**以下 X 个论点仅有 1 个信源支撑,建议补反向验证:**
1. "中国 IP 衍生品市场年增 18%" — 仅艾瑞一家来源
- 建议补查:QuestMobile / 36氪 / 行业上市公司财报
## 五、AI 幻觉 7 项 checklist
- [✅] 精确数字溯源:8/10 个数字已确认来源
- [⚠️] 张冠李戴检查:发现 1 处时期错位
- [✅] 来源链接验证:见 §一
- [✅] 逻辑一致性
- [⚠️] 过度外推检查:1 处用 1 城样本推广全国
- [✅] 时间线正确性
- [❌] 综合幻觉检查:发现 1 处"创造性合并"
## 六、矛盾检测
| 指标 | 段落 A | 段落 B | 差异 |
|---|---|---|---|
| 程序员人数 | 940 万 | 1000 万 | 6.4%(>5% 阈值) |
## 七、行动建议(top 3 风险)
1. **🔴 必须删除**:URL https://... 返回 404,引用此 URL 的论点全部要核实
2. **🟠 必须补源**:"$4.2B 市场规模"无原始出处,疑似 AI 编造
3. **🟡 建议核实**:940 万 vs 1000 万 程序员人数前后矛盾,统一口径
---
> 本报告由 fact-check-cy 自动生成。最终决策由人工拍板。
> 详细方法论参见 [SOP P4 验证质控](~/Documents/think/市场调研SOP-方法论与流程.md)
Tool usage constraints
| Tool | Purpose | Notes |
|---|---|---|
| Read | Read markdown report files | Only read known paths |
| Grep | Extract URLs, numbers, and key arguments | Use regular expression matching |
| Bash | Batch parallel processing, lightweight data cleaning | Avoid unnecessary complex scripts |
| WebFetch | URL health checks, source content retrieval | Every URL must be verified, in parallel batches (5 per batch) |
| WebSearch | Reverse number lookup, CRAAP source background | Chinese-English bilingual; prioritize whitelisted sources |
| Glob | Locate report files in the user's working directory | Only use when the user says "that report from earlier" |
Global constraints
- Human-driven: do not run proactively——unless triggered by the user, or do not proactively recommend a second time for suggestions that were declined during proactive suggestions
- Clearly mark as unverifiable: all ⚠️ / ❌ items must provide "sources recommended for supplementary verification"
- Do not fix the report: only output the risk list and suggestions, do not modify the original text without authorization
- Light tier first: default to light mode; enter deep only when the user explicitly says "deep verification"
- Whitelist weighting: default
whitelist_boost: on, see reference.md §4 for details
Detailed reference
CRAAP 5-dimension scoring rules, complete examples of four-dimensional triangulation verification, 7-item AI hallucination follow-up question templates, must-check source whitelist for MINISO business, complete output report template:
See reference.md