devils-advocate

Devil's Advocate Structured Counter-Argumentation

Turn "playing devil's advocate" from an attitude into a process with sources, auditability, and continuous monitoring.

Why it is designed this way (only by understanding it will you not deviate)

Three empirical findings determine the shape of this skill:

By default, LLMs will please the questioner. Models trained with RLHF systematically tend to cater to the user's already expressed beliefs rather than tell the truth (Sharma et al., ICLR 2024). When analysts directly ask the model "is my argument correct," the answers they get have a structural positive bias. Therefore, the counter-argument role must be explicitly assigned; one cannot expect the model to spontaneously play devil's advocate.
Counter-argumentation without evidence is not only useless but also counterproductive. A classic result in social psychology: role-playing-style "assigned opposition" is less effective than genuine dissent (Nemeth et al., 2001); subsequent research further points out that inauthentic dissent actually makes people cling more firmly to their original judgment (Nemeth, 2018). Therefore, every counter-argument in this skill must be anchored to a specific location in specific material—evidence anchoring is the proxy for "genuine dissent," and freewheeling contrarianism is prohibited.
A multi-agent structure with an explicit counter-argument persona has quantitative gains.The "critic" role shows a 6.4-12.5 percentage point improvement in correlation coefficient over the previous state-of-the-art evaluation method (G-Eval) (DEBATE, ACL Findings 2024); the paper also qualitatively concludes that the counter-argument persona is significantly better than a neutral multi-agent baseline; multiple "neutral" agents evaluating each other may instead reinforce each other's erroneous consensus.

Routine execution of investment arguments does not require loading the two references; when you need to explain the design rationale, have doubts about a certain step, or stress test the engineering mechanism, read Methodological lineage and engineering applicability boundaries. For the engineering mechanism, first read the section "The falsification boundaries of the engineering mechanism" within it, then test along the process below; for the original prompt and schema of the replicated product, see references/linqalpha_reference_implementation.md。

Entry-point judgment

What the user provided	Which path to take
Argument + evidence material (file/directory/URL list)	Full six-step process (below)
Only the argument, no material	First ask the user for material; if the user says there is none → ask the user to choose between two options: ① first do retrieval to build up the material and then come back to run it (if there is a retrieval-type skill, it can be used first) ② deliver only the Step 1 assumption breakdown + Step 5 signal markers (these two steps do not depend on material; at this point there is no risk_flag to filter by, and the signal markers cover all load-bearing assumptions), and clearly state that counter-evidence retrieval was not done
The argument is vague (a piece of prose, an IC memo)	First extract the core assertion per Step 0 and have the user confirm it, then continue
More than 8 assumptions, or the material is extremely large (total over about 200,000 characters / a single file cannot be read to the end)	Use fan-out mode in the main session (see the end of the document); all load-bearing assumptions enter retrieval; when running this skill in a sub-agent, only single-context is possible, so take the top 8 by load-bearing ranking

Step 0 · Input confirmation

Three things; if any one is missing, fill it in first:

Argument: One core claim + the reasoning supporting it. When the user provides a long document, distill it into the form "X will Z because of Y" and ask the user to confirm—if you break down the wrong thesis, everything after that is wasted. When running inside a subagent with no user available to confirm: write out the normalized thesis, place it at the top of both deliverables, and mark it "unconfirmed," then continue.
Evidence inventory: Specific file paths / directories / URLs. Confirm each one is readable; report unreadable ones on the spot, do not silently skip them.
Purpose: Self-check before a meeting, or evaluating someone else's proposal? This determines the tone and level of detail of the human-readable layer.

Step 1 · Break down assumptions

Break the thesis into two categories, numbered uniformly A1, A2, A3…:

Explicit assertions(explicit assertions): those written directly in the thesis.
Implicit assumptions(implicit assumptions): premises that are not written out but must be true for the thesis to hold. This category is the main breeding ground for blind spots—confirmation bias prevents people from seeing what they depend on.

Tag each assumption with two attributes:

Type: factual (can be checked now) / predictive (about the future) / mechanistic (about the causal chain). The type determines what kind of counterevidence Step 2 uses.
Load-bearing(load-bearing, the RAND ABP criterion): If this assumption is overturned, will the conclusion change substantially? If not → downgrade it, do not enter counterevidence retrieval, and list it only in the appendix.

There is no upper limit on enumeration; list them all faithfully; the "8 items" constraint applies to the number thatenter counterevidence retrieval. If a single assumption hides two sub-propositions whose conclusions may be opposite (for example, "a barrier exists" and "the returns from the barrier accrue to one particular company"—the former may survive and the latter may be reversed), split them into two and retrieve for each separately, then rank them together by load-bearing. In single-context mode, when more than 8 items enter retrieval, take the top 8 by load-bearing, and explicitly list the rest as "not retrieved"—better to dig deep into 6 than skim 15; the main-session fan-out mode is not subject to this limit, and all load-bearing assumptions enter retrieval.

Step 2 · Retrieve counterevidence for each assumption

Ask two questions of each assumption (the second is the one most easily missed):

What in the materials directly weakens, contradicts, or introduces uncertainty into it?
If this assumption is true, what evidence should have been seen in the materials but was not?(the core question of ACH—absent evidence is also evidence)

When retrieving, accept sources in order of credibility (adapted from LinqAlpha's SEC > expert interviews > broker reports):

Official filings/announcements/first-hand data → expert interviews and earnings call transcripts → broker/analyst reports → authoritative media reports → aggregated paraphrase/social media (can only serve as leads, not as anchors for counterevidence)

Hard rules:

Every counterargument must carry a citation: source file (or URL) + location (page number/line number/section) + verbatim excerpt. Quotations must be verbatim; use ellipses for omissions; it is forbidden to paraphrase and still use quotation marks.
If none can be found, write "no counterevidence found in the given materials", set citations to empty. This is a valid and informative output—but the final report must note that its meaning is only "the materials do not cover this," not "the hypothesis holds." Exception: when the materials explicitly record that a dedicated search was conducted and nothing was found, this itself is citable evidence ("evidence of absence" is stronger than "absence of evidence"), and citations point to that record.
Each piece of evidence belongs to one primary owner: place it under the hypothesis it most directly refutes; when other hypotheses need it, point to it by citation number; it may appear repeatedly only when it constitutes an independent direct refutation of each of two hypotheses.

Step 3 · Structured findings (audit layer)

By default, write to the current working directory's da_findings.json(if the user specified a location, use the user's; for programmatic processing and review). The top level is anobjectrather than a bare array—the coverage declaration needs a place to live (the original prompt used a bare array; this skill changes it to an object for this reason):

{
  "run_metadata": { "thesis_normalized": "<规范化论点（无用户确认时标「未经确认」）>",
                    "materials": ["<材料清单与各自偏性>"],
                    "coverage": "<枚举 N 条 / 检索 M 条 / 未检索 K 条 / 降级 J 条>" },
  "findings": [ "<每条假设一个条目，结构见下>" ],
  "deferred_assumptions": [ "<降级与未检索的假设及理由>" ]
}

In findings, one entry per hypothesis:

{
  "assumption_id": "A1",
  "assumption": "<该假设的一句话精炼复述>",
  "type": "事实性 | 预测性 | 机制性",
  "counter_argument": "<证据支撑的反方论证，分析师文风，2-3 句>",
  "citations": [ { "doc": "<文件名或URL>", "loc": "<页/行/节>", "excerpt": "<逐字摘录>" } ],
  "rebuttal": "<材料中削弱本条反证的反-反证（如「打断者自己也延期了」），有则填；无则 null>",
  "missing_evidence": "<若假设为真本应看到却没看到的证据；无则 null>",
  "base_rate": "<外部视角：这类断言的历史成立率，有数据才写，无则 null>",
  "risk_flag": "High | Medium | Low",
  "signpost": "<可观测的证伪信号：什么事件/阈值出现说明这条假设正在失效，到哪里看>"
}

risk_flag measures how damaging this counter-evidence is to the argument, not its probability of occurring. Three anchor levels:High = if this counter-evidence holds, the argument's core conclusion no longer holds;Medium = the conclusion still stands, but its magnitude or timeline is significantly impaired;Low = marginal correction. Anti-inflation constraint: if more than half the entries are rated High, first rank all entries by damage against one another and then re-rate them—all High equals no grading at all (in the first live test, the first version rated 6/8 as High).citations at least one (for exceptions see the empty-setting rule in Step 2); strong counter-evidence is often formed by the accumulation of multiple independent facts—list them all rather than picking just one.base_rate is Mauboussin's outside view (a layer LinqAlpha lacks): place the hypothesis into a reference class and ask "when others were in the same situation, how did things turn out"—fill this in only when a reference class can be constructedwithin the given materials(for example, using the correction magnitude of historical forecasts recorded in the materials themselves); if it cannot be supported, honestly leave it empty. Do not fabricate a base rate, and do not start a separate search for this: a separate search would break the test boundary of "argument vs. given materials." If an external base rate is truly needed, explicitly ask the user to supply additional materials.

Step 4 · Human-readable layer rendering (delivery layer)

Structured entries are not shown directly to people (the original actual product was also rendered into a narrative layer for delivery). By default, the human-readable layer is written to the working directory's da_report.md and delivered in the conversation at the same time. Rendering rules:

Opening short paragraph: first give the coverage declaration (which materials were searched, how many hypotheses entered the search, how many were not searched, what biases exist in the composition of the materials—for example meta-issues like "all three research reports are buy ratings"), then in a few sentences point out the 2-3 most damaging pieces of counter-evidence and the weakest side of the argument. Do not give an overall verdict that the argument is right/wrong.
Bodygrouped by theme(the grouping dimension by default follows the argument's own chain of reasoning, one group per link, with several related hypotheses merged under one subheading), not listed mechanically as A1/A2; in the prose, use inline numbers [1] [2] to attach citations—[n] is numbered in the order in which the citation first appears in the body.
Each argument is 2-3 sentences, in the style of an institutional analyst: factual, restrained, focused on substantive risks (competition, regulation, profit margins, technological substitution), with no exclamation marks and no rhetorical questions.
Three lists at the end:References(the source corresponding to each [n] number and the verbatim excerpt, with each entry noting the hypothesis number A1/A2 it refutes),surviving hypotheses(Entries for which no counterevidence was found after retrieval are noted as "not found ≠ holds"),Coverage Appendix(Downgraded non-load-bearing assumptions + assumptions not retrieved due to exceeding limits, providing detailed support for the coverage statement at the beginning).

Step 5 · Signal Indicator Checklist

The final section of the report: one line per High/Medium assumption — "what to watch, where to look, what reading counts as an alert" (in no-material mode there is no risk_flag to filter by, so all load-bearing assumptions are covered). The fragility of assumptions changes over time; this step extends the one-off critique into sustainable monitoring, which is where this skill's long-term value lies — users are advised to incorporate it into their own monitoring routine.

Red Lines

Do not judge the overall rightness or wrongness of the argument on the user's behalf, and do not give buy/sell recommendations — the output of this skill is a "stress checklist"; decisions are left to humans.
The strength of counterevidence is limited by the evidence: if the material only supports "introducing uncertainty," it must not be written as "falsification."
Tone calibration: factually neutral throughout. The devil's advocate is a procedure, not an emotion.
Coverage honesty: the report states at the beginning which materials were retrieved, how many assumptions entered retrieval, and how many were not retrieved.

What this skill cannot do (written for users, and also for the executing model)

It tests "argument vs. given materials," not "argument vs. the world." The materials themselves may be biased (for example, if they are all bullish research reports), so counterevidence will naturally be sparse — the report should point out the bias in the composition of the materials and, when necessary, suggest what types of materials to supplement.
It cannot replace a real opposing position holder. Nemeth's results also constrain AI: evidence anchoring alleviates the "role-playing is ineffective" problem, but a human opponent with real money in an opposing position is still a stronger test (this is the logic behind Buffett inviting short sellers to the shareholders meeting).
A single run is a snapshot. The fragility of assumptions changes over time — which is why the signal indicator checklist is where the long-term value lies, and users are advised to incorporate it into their own monitoring routine.

Fan-out Mode (available only in the main session)

When the volume of materials or the number of assumptions is large, the main session parallelizes as follows:

The main session completes Step 0-1 (assumption decomposition must be done in the main context — it requires a complete understanding of the argument).
Dispatch one sub-agent per assumption (or per 2-3 assumptions): the prompt gives "the original assumption text + material path list + the two questions of Step 2 + the JSON structure of Step 3 and the three-tier anchoring of risk_flag (inlined verbatim) + verbatim quotation and no-fabrication rules," and specifies each one's absolute output path (under the working directory da_findings_A<编号>.json), requiring them to write to disk first and then return a summary.
After the main session collects everything, merge into da_findings.json: deduplicate (when the same evidence is cited by multiple assumptions, attribute it to the one with the higher risk_flag, and change the other entries to point to the same citation number), recheck risk_flag against the three-tier anchoring, then do the rendering and signal indicators of Step 4-5.
When this skill is run inside a sub-agent, no further agents may be dispatched — fall back to single-context mode.

Combination with Other Skills

When materials are insufficient, first use retrieval/industry research skills (such as daymade-sector-research from the same suite) to build up the evidence base, then come back and run this process — this skill assumes the materials are already in hand.
The natural next step after stress testing is complete: incorporate the signal indicator checklist into the user's monitoring routine, or run another round on the revised argument 2.0 (versioned comparison: which counterevidence was absorbed and which was rebutted). Suggest as needed; do not push hard.

Design Basis (Quick Reference)

Design Element	Source
Four-step process, explicit/implicit assumption classification, mandatory citation, risk grading, dual-layer output	LinqAlpha Devil's Advocate (production prompt and schema publicly released in an AWS official technical blog, 2026-02)
Definition of assumptions that are "unstated but must be true," with item-by-item questioning of "why must it hold"	CIA Key Assumptions Check（Tradecraft Primer, 2009）
Falsification first, "evidence that should have been seen but was not"	Analysis of Competing Hypotheses（Heuer / CIA Primer）
Load-bearing assumptions, signposts	RAND Assumption-Based Planning（1993/2002）
Outside view / base rate layer	Mauboussin & Callahan, The Base Rate Book（2016）
Necessity of evidence anchoring (role-play-style dissent is ineffective)	Nemeth et al.（2001）
An explicit opposing-view persona outperforms a neutral multi-agent setup	DEBATE (ACL Findings 2024); Du et al. multi-agent debate (2023)