Assess computational health, predictive adequacy and scientific validity separately, matching each check to the claim it can support.
Workflow
- Identify the target and output. Establish the model, observations,
algorithm, independent chains, warmup and retained draws. A
posteriorgroup does not prove that draws came from MCMC; an observation-free model samples a prior target. Preserve chains and draw order, and save inference before costly postprocessing. Establish whether this is exploratory output or intended to support reportable inference: a rough fit may guide debugging without being accepted as an accurate posterior. - Check exploration and precision. Access
idata["posterior"]andidata["sample_stats"]. Count HMC divergences by chain; inspect rank-normalized split R-hat, bulk/tail ESS and MCSE for the actual estimands, including relevant latent coordinates. R-hat below 1.01 and ESS above 400 are screening heuristics, not guarantees; choose precision requirements in scientific units. - Inspect the plots. Look for drifting/stuck chains, rank imbalance and divergence clusters. Use energy/BFMI, autocorrelation and ESS evolution where appropriate. Missing or nonfinite diagnostics are unresolved, not zero; HMC statistics may be inapplicable for other algorithms.
- Address causes before adding draws. Investigate scaling, gradients,
constraints, identifiability and parameterization. Non-centering often helps
weakly informed hierarchies; centering can suit strong data. Higher
target_accept, thinning or dropping bad chains cannot certify a repair. - Criticize predictions separately. Generate replicated observations and inspect task-relevant discrepancies, conditional calibration and uncertainty. Distinguish latent means from noisy new observations. In-sample PPCs are not held-out predictive validation.
- Evaluate the declared prediction target. Compute pointwise log likelihood explicitly, check PSIS reliability before LOO/ELPD comparisons, and keep the same observations in the same order. Use grouped holdouts for new groups and past-only training for forecasts. Diagnose high Pareto k rather than hiding it.
- Interpret uncertainty honestly. Report paired ELPD uncertainty and practical relevance, not only ranks. Stacking weights are neither model probabilities nor equivalence tests. Never exponentiate an ELPD difference as a Bayes factor. Adaptive model revisions and repeated CV comparisons can overfit selection. Explain consequential revisions and compare substantive inferences across viable alternatives, not only a winning score. For an action recommendation, propagate uncertainty through stated loss/utility and constraints, or hand off the inference to the decision maker without inventing their preferences.
References
- Diagnostics and predictive checks — a complete diagnostic example, DataTree APIs, interval/MCSE semantics, plots, regression, counts, survival and nested chains.
- Predictive evaluation and model comparison — LOO, high-k remedies, predictive metrics, stacking and validation design.
Examples use the modern ArviZ package family; version-specific cautions are identified in the references. This skill does not require another skill.