Individual pieces of evidence are unreliable. Synthesis across sources — weighting by quality, identifying convergence and divergence, resolving conflicts — produces conclusions worth acting on. Conclusion quality is bounded by the quality of the weakest evidence it leans on most heavily.
When to Use
- Multiple sources need integration into a single conclusion
- causal-inference produced experimental findings to weigh against other data
- learning-strategy produced summaries from multiple sources
- Sources disagree and the user needs to know what to believe and why
- A decision requires a "state of the evidence" assessment
- deep-document-processor extracted findings from several documents
Core Methodology
Step 1: Frame the Synthesis Question
State the specific question to prevent scope creep and keep synthesis decision-relevant.
- Question: what specific claim, recommendation, or understanding should emerge?
- Decision context: what will be done with the conclusion? (Stakes determine rigor.)
- Scope boundaries: inclusion criteria — time period, domain, methodology type, minimum quality threshold
A synthesis without a clear question becomes a literature dump.
Step 2: Inventory All Evidence Sources
List every piece with provenance. Accept inputs from upstream skills:
- From causal-inference (Contract J): primary experimental data; includes quality grade, causal claim with confidence, limitations
- From learning-strategy (Contract L): varies (book, paper, expert opinion, course); per-source quality grade, key findings, gaps
- From deep-document-processor or direct input: document/dataset/report; assess quality during this step
For each source, record:
- Source identifier: name, author, date
- Source type: primary data (experimental/observational), systematic review, expert opinion, anecdote, proxy metric, model output
- Key finding: main claim or data point
- Methodology: how the finding was produced (brief)
- Relevance: how directly it addresses the synthesis question
Step 3: Grade Evidence Quality
Use the GRADE-adapted framework:
| Grade | Criteria | Typical sources |
|---|---|---|
| High | Well-designed study, adequate sample, consistent results, directly relevant | RCTs, large observational studies with controls, systematic reviews |
| Moderate | Sound methodology with some limitations, moderate inconsistency, mostly relevant | Smaller studies, quasi-experiments, well-reasoned expert analysis |
| Low | Observational without controls, small samples, significant limitations, indirect | Case studies, uncontrolled observations, adjacent-domain analogies |
| Very Low | Expert opinion without data, anecdote, biased source, tangential | Individual opinions, single anecdotes, motivated reasoning, outdated |
Upgrade: large effect size, dose-response, all plausible confounders would reduce the effect (conservative bias).
Downgrade: risk of bias, inconsistency across studies, indirectness (different population/setting), imprecision (wide CIs), publication bias.
A "High" grade you can't justify is really "Moderate."
Step 4: Identify Convergent Findings
Claims supported by multiple independent sources are the strongest signal.
- Group sources by claims they support
- Convergence from diverse methodologies (experiment + observation + expert agreeing) is stronger than three experiments by the same team
- Check whether convergent sources are truly independent or share data, assumptions, authors
- Weight by source quality — three High-grade converging beats five Very Low agreeing
Record: finding, supporting sources, combined quality, confidence.
Step 5: Analyze Divergent Findings
Diagnose disagreement before choosing sides:
- Methodology differences: different designs legitimately produce different results (survey vs. experiment measure different things)
- Context differences: enterprise software finding may not apply to consumer apps; 2019 result may not hold in 2026
- Scope differences: sources may measure different aspects of the same phenomenon — not actually contradicting
- Genuine disagreement: after accounting for the above, sources still disagree. Most informative — the question is harder than it looks, or one side has an unidentified flaw.
For each divergence: state the conflicting claims, diagnose likely cause, assess which to weight more (and why), flag remaining uncertainty.
Step 6: Identify Evidence Gaps
What evidence is missing that would matter?
- Questions the synthesis needs answered but no source addresses
- Quality gaps — only Very Low or Low evidence on the topic
- Perspective gaps — all sources share methodology, geography, or theoretical lens
- Recency gaps — all evidence predates a significant context change
For each gap: what evidence would fill it, importance, feasibility (feeds back to experimental-design or learning-strategy).
Step 7: Produce the Synthesis Conclusion
Integrate into a confidence-weighted conclusion:
- Primary conclusion: plain language; lead with what's most supported
- Confidence level: High (multiple High-grade converge, no significant divergence) / Moderate (some convergence with caveats) / Low (limited or conflicting) / Very Low (mostly opinion or indirect)
- Key caveats: conditions where the conclusion might not hold
- What would change the conclusion: pre-commit to revision triggers (synthesis equivalent of epistemic-mapping's mind-change conditions)
The conclusion should be decision-ready — downstream skills like decision-synthesis can take it as a criterion with the confidence level indicating weight.
Output Format
🔍 Synthesis Question
- Question: [What does the evidence say about X?]
- Decision context: [What will be decided based on this synthesis?]
- Scope: [Inclusion criteria — time, domain, methodology, quality floor]
📋 Evidence Inventory
| # | Source | Type | Key Finding | Quality | Relevance |
|---|---|---|---|---|---|
| 1 | [Source] | [Experimental/Observational/Review/Expert/Anecdote] | [Main claim] | [High/Moderate/Low/Very Low] | [Direct/Indirect] |
✅ Convergent Findings
- Finding 1: [Claim supported by multiple sources]
- Supported by: [Source #s], combined quality: [grade]
- Independence: [Are sources truly independent?]
⚠️ Divergent Findings
- Disagreement 1: [Source A says X, Source B says Y]
- Likely cause: [Methodology/Context/Scope/Genuine]
- Resolution: [Which to weight more, why]
- Remaining uncertainty: [What we still can't resolve]
🕳️ Evidence Gaps
- [Gap 1]: [What's missing, why it matters, how to fill it]
🏆 Synthesis Conclusion
- Finding: [Plain-language conclusion]
- Confidence: [High/Moderate/Low/Very Low]
- Key caveats: [Conditions where this might not hold]
- Would change conclusion if: [Pre-committed revision triggers]
Thinking Triggers
- "Am I weighting this source because it's good, or because it agrees with what I already believe?"
- "If I removed the weakest source, would the conclusion change? If so, it's resting on thin evidence."
- "Are these sources truly independent, or do they share data, methods, or authors?"
- "What evidence, if it existed, would make me conclude the opposite?"
- "Am I treating absence of evidence as evidence of absence?"
- "Would a critic point to a source I'm underweighting?"
Common Traps
- Vote counting: "3 say yes, 1 says no, so yes." Quality and effect size matter more than count. One rigorous experiment outweighs five poorly designed surveys.
- Confirmation weighting: Grading sources that support your prior as higher quality. Grade by methodology, not by agreement with your hypothesis.
- Narrative smoothing: Presenting synthesis as more coherent than evidence warrants. Real evidence is messy — acknowledge it.
- Anchoring on the first source: First evidence read disproportionately shapes the conclusion. Read challenging counter-evidence early.
- Ignoring base rates: A surprising finding from one source doesn't override strong prior evidence. Update proportionally.
Example Applications
| Trigger | Application |
|---|---|
| "A/B test and survey data disagree" | Inventory both, grade quality (experiment usually higher than survey for behavioral claims), diagnose divergence (stated vs. revealed preference), synthesize with appropriate weighting |
| learning-strategy produced summaries from 6 papers on microservice migration | Inventory per Contract L, grade each, identify convergent strategies, flag gaps (no papers from companies at our scale), produce confidence-weighted recommendation |
| "Three team members have different opinions on root cause" | Treat each as expert opinion (Very Low–Moderate depending on backing), look for convergence, identify what data would resolve, recommend the investigation that would upgrade quality |
| causal-inference found "probable" causation + two industry reports | Integrate experimental evidence (Contract J) with reports, assess independence, note timing relative to experiment, produce combined confidence level |