Engagement is the metric that arrives free in your event stream, which is why most products prove only that people came back. This skill is about the other three quarters of the claim — did they learn it, can they use it, and does anyone outside your company agree — plus the instrument design that makes those answers survive contact with a skeptic.
The organizing idea: a number you can move at will cannot support a claim about the world. Every node here is a way to hand part of the grading to something you do not control — an external standard, an independently authored item pool, your own population in aggregate, the shape of a distribution.
Decide what to measure
- [[references/four-axes-that-do-not-substitute]] — enjoyment, mastery, real-task application and external standard are four independent questions, each needing its own method; and none of them is one number.
- [[references/borrow-the-instrument-you-do-not-control]] — run your users through the incumbent's own test, compare against a published study of the incumbent's users, and report the axis where you lose.
Design the instrument
- [[references/build-the-item-pool-independently-of-the-content]] — questions drawn from the material you shipped measure exposure; an independently authored pool is your holdout set.
- [[references/embed-a-pre-test-and-post-test-in-the-product]] — put the measurement in the path as a gate, ask about the next segment before you teach it, and keep the scores private.
- [[references/thin-sample-many-people-to-grade-the-curriculum]] — a handful of questions per person, aggregated across everyone, grades your content instead of your users and hands you a ranked build list.
Score it well
- [[references/adaptive-is-shorter-not-easier]] — selecting each item near the running estimate makes every answer informative, so the same standard is reached in far fewer items.
- [[references/partial-credit-beats-binary-scoring]] — binary grading discards the gradations that separate adjacent ability levels, and the symptom is a person who retests far from where they landed before.
- [[references/check-the-distribution-not-the-score]] — calibration is read off the shape of the curve; a ceiling means the instrument stopped separating people exactly where the decisions get made.
Where the boundary sits
Choosing the next item for a live person in a session — targeting the band between bored and lost, modelling item and user jointly — belongs to [[../duo-difficulty-calibration/SKILL]]. Adaptive machinery appears in both. The split is what the output is for: a score, or evidence about the product, routes here; a session assembled for one individual routes there.
Sibling skills
- [[../duo-experimentation/SKILL]] — designing the test that tells you whether a change worked, and the ladder of evidence behind it.
- [[../duo-difficulty-calibration/SKILL]] — picking the right next item for a specific person, rather than scoring the product.
- [[../duo-product/SKILL]] — what to do with an efficacy result once you have one, and how it disciplines the roadmap.
- [[../duo-learner-motivation/SKILL]] — why people stay engaged at all, which is the axis efficacy measurement cannot substitute for.
Sources
Drawn from blog.duolingo.com — efficacy research framework, published proficiency studies, checkpoint and learning quizzes, the placement test, and the Duolingo English Test. Every node carries its own dated citation.