SuperAPI AI Directory

fact-check-cy

General & other

View repository

Updated 10-15 20:30

About this skill

Research report fact-checking skill

The last line of defense before research deliverables are handed over. The error rate of AI-generated URLs is about 60%, and precise numbers often have "creative merging" hallucinations - this skill automatically executes the four core methods of SOP P4 (SIFT / CRAAP / four-dimensional triangulation / 7-item AI hallucination checklist), outputs a risk list and action recommendations, and a human makes the final call on how to revise the report.

Core boundaries

  • Do: verify URL authenticity, number traceability, source evaluation, triangulation, contradiction detection, output a risk list
  • Do not: automatically fix the report (only output recommendations, humans decide how to revise); redo research (that is deep-research's job)

Configuration parameters

Users can specify via natural language, or use default values:

Parameter Value Default Description
mode light / deep light Light mode 5 minutes / deep mode 15-20 minutes (adds all 5 CRAAP dimensions + four-dimensional triangulation)
target full / point full Full report check / single-point data check
whitelist_boost on / off on Enable mandatory-source whitelist weighting (iResearch, QuestMobile, Juliang Suanshu, etc.)
language cn / en / both both Source language preference

Example triggers:

  • "Check this research report" (full + light, default)
  • "Deep-check this report" (full + deep)
  • "Is this 9.4 million figure real" (point)

Execution flow (5 steps)

Step 1: Parse input

Determine the user input type:

  1. Full report check(target: full):

    • User provides a markdown file path → use Read to read
    • User pastes report content → process directly
    • User says "that report just now" → extract the most recent report text from the conversation history
  2. Single-point check(target: point):

    • Extract the "data + source" combination from the statement
    • For example: "China has 9.4 million programmers, from the Ministry of Industry and Information Technology" → data = 9.4 million / subject = Chinese programmers / source = Ministry of Industry and Information Technology

Output a structured checklist of verification tasks (URL list, number list, core source list) for parallel processing in subsequent steps.

Step 2: URL health check

This is the highest-priority step - AI-hallucinated URLs account for about 60%, so if you do not verify them, there will almost certainly be problems.

Method:

  1. Use Grep or Bash regex toextract all URLs from the report:

    • Markdown links:\[.*?\]\((https?://[^)]+)\)
    • Plain-text URLs:https?://[^\s\)\]"',。、;]+
    • After deduplication, obtain the URL list
  2. In parallelrun WebFetch:

    • on each URL to verify the HTTP status code (200 = pass, 404/403/5xx = fail)
    • Retrieve the page title and body summary
    • Compare keyword overlap with the citation context (50 characters before and after)
  3. Output three lists:

    • ✅ Accessible and relevant: status 200, content overlap with context ≥3 keywords
    • ⚠️ Accessible but content does not match: Status 200, but the content is unrelated to the context (the most covert AI hallucination)
    • ❌ Invalid: 404 / 403 / 5xx / DNS resolution failure

Key constraints: When there are many URLs (>20), process them in parallel batches (5 per batch) to avoid sending too many WebFetch requests at once.

Step 3: Precise number traceability

Prevent the AI from "creatively merging" numbers that do not exist.

Approach:

  1. Use Grep to extract all "precise number" patterns from the report:

    • Amounts:\$\d+(\.\d+)?[BMK]? / ¥\d+(\.\d+)?[万亿]?
    • Percentages:\d+(\.\d+)?%
    • Large numbers:\d+(,\d{3})+ or \d+\s*(万|亿|百万|千万|million|billion)
    • Growth rates:(YoY|同比|环比|CAGR)\s*[+-]?\d+(\.\d+)?%
  2. For each number, findthe nearest cited source(the URL or institution name in the preceding or following paragraphs).

  3. Use WebSearch for reverse lookup: use "number + subject + time" as the query to see whether the original source can be found.

    • For example:"940 万" 中国程序员 工信部 2024
  4. Label each number:

    • ✅ Original text source found: This precise number can be seen in the original source
    • ⚠️ Fuzzy match only: Related data can be found, but it is not the exact same number (difference < 10%)
    • ❌ Not found in original text: Cannot be found in the cited source, a strong indicator of AI hallucination risk

Step 4: CRAAP source evaluation (only the deep mode executes the complete 5 dimensions)

For eachcore source(sources cited ≥2 times), perform a 5-dimension evaluation.

Automatic scoring logic(see reference.md §1):

Dimension Scoring basis
Currency Timeliness WebFetch to get the publication date → time since then <6 months = 5 / 6-12 months = 4 / 1-2 years = 3 / 2-3 years = 2 / >3 years = 1
Relevance Relevance Degree of keyword overlap between the source content and the cited argument (≥5 = 5 / 4 = 4 / 3 = 3 / 2 = 2 / 1 = 1)
Authority Authority Domain lookup whitelist (iResearch / QuestMobile / Juliang Suanshu / 36Kr / listed company financial reports → 5 / mainstream media → 3-4 / unknown blogs → 1-2)
Accuracy Accuracy Whether the argument is supported by ≥2 independent sources (≥3 sources = 5 / 2 sources = 4 / 1 source = 2)
Purpose Purpose Whether it is commercial/marketing/PR (academic/official reports = 5 / neutral media = 4 / industry analysis = 3 / company PR = 2 / stakeholder statements = 1)

Each core source receives a total score of 0-25.Red flag rules: Any dimension ≤2 → mark red (detailed in reference.md §1).

Light mode skips this step, performing only the most basic ≥2 source cross-check.

Step 5: Triangulation + contradiction detection + 7-item AI hallucination checklist + output report

5a. Triangulation (key arguments)

  1. Automatically identify "conclusive arguments" in the report:
    • Bold paragraphs (\*\*...\*\*)
    • containing keywords such as "research shows", "data shows", "survey finds"
    • paragraphs with high number density
  2. For each argument, count the sources supporting itNumber of independent sources(deduplicated by domain)
  3. Markers:
    • ✅ ≥2 independent sources → triangulation verified
    • ⚠️ 1 source → strong prompt to add a reverse source
    • ❌ 0 sources → purely AI-generated, must be deleted or sources added

5b. Additional work for the deep tier: four-dimensional triangulation verification

See reference.md §2 for implementation details covering 4 categories (data sources / methodology / investigators / theory).

5c. AI hallucination 7-item checklist

Mark each item as ✅ / ⚠️ / ❌, see reference.md §3:

  1. Precise number traceability——whether each number can be traced back to its original source
  2. Misattribution check——whether data has been incorrectly attributed to other companies/periods/markets
  3. Source link verification——already completed in Step 2
  4. Logical consistency——whether the data/conclusions in different sections are self-consistent
  5. Over-extrapolation check——whether limited data has been over-generalized
  6. Timeline correctness——whether the order of events and causal chronology are reasonable
  7. Comprehensive hallucination check——whether the AI "creatively merged" data from multiple sources into conclusions that do not exist

5d. Contradiction detection

When the same metric appears multiple times in different sections of the report, compare the values. Difference >5% → mark as a contradiction point.

5e. Output verification report

Generate a markdown report in the format below, and output it directly (do not archive it unless the user explicitly requests it).

Output report format

# 📋 事实核查报告

> **模式**:{light | deep}    **覆盖**:X 个 URL / Y 个数据点 / Z 个核心信源
> **核查时间**:YYYY-MM-DD HH:MM

## 总览

- ✅ 通过:X 项
- ⚠️ 待补强:X 项
- ❌ 不通过:X 项
- **综合可信度评分:X / 100**

## 一、URL 健康度(X / Y 个)

| URL(截断) | 状态 | 上下文匹配 | 风险 |
|---|---|---|---|
| https://... | ✅ 200 | ✅ 相关 | — |
| https://... | ❌ 404 | — | **必须删除引用** |
| https://... | ✅ 200 | ⚠️ 不匹配 | **疑似 AI 幻觉,必须核实** |

## 二、数据点溯源(X / Y 个)

| 数字 | 引用源 | 反查结果 | 风险等级 |
|---|---|---|---|
| 940 万程序员 | 工信部 2024 | ✅ 知乎/EET-China 多源确认 | 低 |
| $4.2B 市场规模 | 报告未注明 | ❌ 反查无结果 | **高(疑似幻觉)** |

## 三、CRAAP 评估(核心信源,仅 deep 模式)

| 信源 | C | R | A | A | P | 总分 | 红旗 |
|---|---|---|---|---|---|---|---|
| 艾瑞咨询 2025 报告 | 5 | 5 | 5 | 4 | 4 | 23 | — |
| 某不知名博客 | 3 | 4 | 1 | 1 | 2 | 11 | 🚩 权威性 / 准确性 |

## 四、三角验证缺口

**以下 X 个论点仅有 1 个信源支撑,建议补反向验证:**

1. "中国 IP 衍生品市场年增 18%" — 仅艾瑞一家来源
   - 建议补查:QuestMobile / 36氪 / 行业上市公司财报

## 五、AI 幻觉 7 项 checklist

- [✅] 精确数字溯源:8/10 个数字已确认来源
- [⚠️] 张冠李戴检查:发现 1 处时期错位
- [✅] 来源链接验证:见 §一
- [✅] 逻辑一致性
- [⚠️] 过度外推检查:1 处用 1 城样本推广全国
- [✅] 时间线正确性
- [❌] 综合幻觉检查:发现 1 处"创造性合并"

## 六、矛盾检测

| 指标 | 段落 A | 段落 B | 差异 |
|---|---|---|---|
| 程序员人数 | 940 万 | 1000 万 | 6.4%(>5% 阈值) |

## 七、行动建议(top 3 风险)

1. **🔴 必须删除**:URL https://... 返回 404,引用此 URL 的论点全部要核实
2. **🟠 必须补源**:"$4.2B 市场规模"无原始出处,疑似 AI 编造
3. **🟡 建议核实**:940 万 vs 1000 万 程序员人数前后矛盾,统一口径

---

> 本报告由 fact-check-cy 自动生成。最终决策由人工拍板。
> 详细方法论参见 [SOP P4 验证质控](~/Documents/think/市场调研SOP-方法论与流程.md)

Tool usage constraints

Tool Purpose Notes
Read Read markdown report files Only read known paths
Grep Extract URLs, numbers, and key arguments Use regular expression matching
Bash Batch parallel processing, lightweight data cleaning Avoid unnecessary complex scripts
WebFetch URL health checks, source content retrieval Every URL must be verified, in parallel batches (5 per batch)
WebSearch Reverse number lookup, CRAAP source background Chinese-English bilingual; prioritize whitelisted sources
Glob Locate report files in the user's working directory Only use when the user says "that report from earlier"

Global constraints

  • Human-driven: do not run proactively——unless triggered by the user, or do not proactively recommend a second time for suggestions that were declined during proactive suggestions
  • Clearly mark as unverifiable: all ⚠️ / ❌ items must provide "sources recommended for supplementary verification"
  • Do not fix the report: only output the risk list and suggestions, do not modify the original text without authorization
  • Light tier first: default to light mode; enter deep only when the user explicitly says "deep verification"
  • Whitelist weighting: default whitelist_boost: on, see reference.md §4 for details

Detailed reference

CRAAP 5-dimension scoring rules, complete examples of four-dimensional triangulation verification, 7-item AI hallucination follow-up question templates, must-check source whitelist for MINISO business, complete output report template:

See reference.md