Overview
treatment-guideline-appraisal teaches structured critical appraisal of clinical practice guidelines: assessing methodological quality (AGREE II), understanding evidence-grading systems (GRADE), spotting conflicts of interest and panel bias, and deciding whether a guideline deserves to change practice. Guidelines carry authority; this skill is about checking whether they've earned it.
When to use
- Deciding whether to adopt a new guideline in practice or policy.
- Appraising guidelines with AGREE II: scope, stakeholder involvement, rigor, clarity, applicability, editorial independence.
- Understanding GRADE: quality of evidence vs strength of recommendation.
- Detecting industry influence, panel stacking, and evidence cherry-picking.
- Comparing conflicting guidelines on the same topic.
- Developing trustworthy local protocols from guideline evidence.
Core concepts
- AGREE II domains. (1) Scope and purpose: is the clinical question clear? (2) Stakeholder involvement: were patients and relevant disciplines included? (3) Rigor of development: systematic search, evidence selection criteria, explicit link between evidence and recommendations. (4) Clarity: are recommendations specific and unambiguous? (5) Applicability: implementation tools, resource implications, audit criteria. (6) Editorial independence: funding and conflicts disclosed and managed. Score each; rigor and independence weigh heaviest.
- GRADE. Separates quality of evidence (high/moderate/low/very low — based on study design, risk of bias, inconsistency, indirectness, imprecision, publication bias) from strength of recommendation (strong/weak — based on balance of benefits/harms, values, resources). A strong recommendation can rest on low-quality evidence (and vice versa) — read both, not just the recommendation.
- Conflicts of interest. Financial ties to manufacturers of recommended products are common on guideline panels. Check: disclosed COIs, proportion of panel with ties, whether conflicted members voted on related recommendations, and funding of the guideline itself. A guideline recommending a sponsor's drug on weak evidence deserves skepticism proportional to the ties.
- Panel composition bias. Specialty-dominated panels recommend their specialty's interventions (surgeons recommend surgery). Trustworthy guidelines include methodologists, generalists, and patient representatives — and use formal consensus methods (Delphi, nominal group) rather than eminence.
- Evidence selection. Was the search systematic and current? Are key trials missing or mischaracterized? Check whether the guideline cites the trials you'd expect and whether effect sizes are reported as absolute (not just relative) benefits.
- Recommendation wording. "We recommend" (strong) vs "we suggest" (weak/conditional) in GRADE language. Vague verbs ("consider," "may be used") often signal weak evidence dressed as guidance — treat them as such.
- Conflicting guidelines. When guidelines disagree, compare: recency of evidence review, panel independence, GRADE ratings, and whether disagreement is about evidence or about values (e.g. different harm tolerance). Disagreement about values is legitimate; disagreement from cherry-picked evidence is not.
- Living vs static. Evidence moves; guidelines decay. Check the update plan and the date of the last evidence search — a 2019 guideline applied in 2026 needs a delta review.
Practical workflow
- Scope check. Does this guideline address your clinical question and population? (AGREE II domain 1.)
- Independence screen. Funding source, panel COIs, methodologist involvement. Fail here = read with maximum skepticism.
- Rigor appraisal. Systematic review methods, evidence tables, explicit evidence-to-decision process. (AGREE II domain 3.)
- GRADE extraction. For each key recommendation: evidence quality, recommendation strength, and the stated rationale. Note where strong recommendations rest on low-quality evidence.
- Effect-size reality check. Convert relative effects to absolute benefits/harms and NNT/NNH in your population's baseline risk.
- Applicability. Resources, workflow fit, patient values in your setting. (AGREE II domain 5.)
- Decision. Adopt, adapt (document modifications), or reject — with reasons recorded. Set a review date.
Common pitfalls
- Treating all guidelines as equally authoritative (eminence-based vs evidence-based).
- Ignoring COI disclosures — or assuming disclosure alone manages bias.
- Confusing strong wording with strong evidence.
- Relative-risk framing hiding trivial absolute benefits.
- Outdated guidelines applied as current standard.
- Specialty-panel recommendations adopted without considering alternatives.
- Implementing without adaptation to local resources and patient values.