SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillslight-result-analysis

light-result-analysis

Agents & models

View repository

Updated 10-16 06:23

About this skill

In-Depth Analysis of Results and Data

Levels of Analysis (Progressively Deeper)

  1. Description: Metric summary, distribution, and gap versus baseline, with error bars/confidence intervals.
  2. Explanation: Why is this the result? Attribute it to which component of the method (combined with ablation).
  3. Diagnosis: Which results prove the innovation is effective, and which instead expose problems or contradictions.
  4. Insight: Patterns that can become paper highlights; unexpected findings; interpretability evidence (SHAP/feature importance/attention).
  5. Action: Which anomalies need investigation, which conclusions need additional experimental validation, and which cannot be overclaimed.

Must-Check List

  • Significance:≥5 random seeds; under limited compute ≥3 and must be explicitly noted in the report(same standard as m05 experimental design), report mean ± standard deviation; for two groups use Welch t (does not assume equal variance), for non-normal use Mann-Whitney U, for multiple groups use ANOVA+Tukey, for proportions use two-proportion z-test. The trio of p-value + effect size (Cohen's d) + confidence interval, do not report only p. For multiple comparisons use BH-FDR correction (multipletests(method="fdr_bh")）。
  • Paired design identification (determine first, then choose the test): Is the method evaluated onthe same set of units(the same batch of seeds/CV folds/test samples)? If yes → usepaired t / Wilcoxon signed-rank(higher power, removes between-unit variance), not independent Welch/Mann-Whitney; mistaking it as independent will underestimate significance.analyze_results.py --paired-by seed automatically aligns pairs by shared columns, uses d_z for paired effect size and bootstrap for the difference CI (see the "Paired Design Identification" section in references).
  • Slice analysis (to prevent blind spots in aggregate metrics): Overall metrics can mask subgroup failures (overall 85% but only 50% for a certain class). Slice by category/domain/difficulty/sensitive attribute/time period, recompute metrics per sliceand include sample size n, compare against the overall to find anomalous slices; for multiple slice comparisons apply BH-FDR; do not draw strong conclusions from small-n slices (see the "Slice Analysis Protocol" section in references; the fairness dimension relates to a10).
  • Consistency: Is it stable across datasets/settings? Where is it anomalous?
  • Ablation self-consistency: Does performance indeed drop when a component is removed? Is the direction correct? Use anova_lm or regression coefficients to see whether component contributions are significant.
  • Failure cases: Error analysis to find systematic bias; use SHAP (beeswarm/bar) to see which features the model actually relies on and whether that is reasonable.
  • Fairness: Whether comparisons use the same settings, same compute, and same data, to avoid unfair comparisons.
  • Overfitting/leakage: Whether train/val/test gaps are abnormal; check for excessively high feature-label correlation or temporal leakage (deepchecks data_integrity / for drift use Evidently DataDriftPreset).

Tools and Specific Usage

  • Quick health check：ProfileReport(df, minimal=True).to_file(...) one-click output of distribution/missingness/correlation/alerts; deepchecks data_integrity().run(Dataset(df,label=...)) checks for leakage/duplicates/single-value columns.
  • Statistical inference: statsmodels. Regression smf.ols("y~x1+x2",data).fit().summary() Give coefficients/p/R²/AIC (use sm.add_constant to add an intercept for OLS; the formula interface adds it automatically); ANOVA anova_lm(model, typ=2); test in statsmodels.stats（ttest_ind / proportions_ztest / multipletests / het_breuschpagan）。
  • Interpretability: SHAP, see the ready-to-use script scripts/explain_shap.py(one-click generation of the three plots beeswarm/bar/waterfall). For tree models use shap.TreeExplainer(fast), universal fallback KernelExplainer(slow, the background set must be sampled);shap.plots.beeswarm for global direction,bar for importance ranking,waterfall for decomposing a single sample. SHAP reflects the model, not causality.
  • Static publication figures: matplotlib object-oriented interface fig,ax=plt.subplots(layout="constrained")，savefig(dpi=300,bbox_inches="tight") saves PDF/SVG vectors; seaborn axes-level functions (boxplot/heatmap/barplot, barplot by default carries 95% CI error bars) can be embedded in ax, while figure-level ones (relplot/catplot) come with faceting built in. For color schemes use colorblind-friendly colormaps such as viridis, avoid jet.
  • Interactive/exploratory figures：plotly express(px.scatter(...,color=,facet_col=) → write_html) or altair(alt.Chart(df).mark_point().encode(x="a:Q",...)), for appendices/supplementary materials.
  • Drift monitoring：Evidently Report(metrics=[DataDriftPreset()]).run(reference_data=ref,current_data=cur)(API-version sensitive, confirm the version first).
  • Relationship analysis: networkx computes betweenness/pagerank centrality and louvain communities, for citation networks/feature relationships/ablation dependencies.
  • Tracking and compilation: for Jupyter Notebook, Restart&Run All before delivery to ensure reproducibility,nbconvert generates reports; compile multiple experiment groups into a site with Jupyter Book(_config.yml+_toc.yml → jupyter-book build)。

Deliverables

  1. Results analysis report: each finding pairs "phenomenon→cause→evidence→significance for the paper".Standard artifact: claim↔evidence correspondence table written to disk as claim_evidence_table.md(the handoff artifact passed to m07/m09, naming per CONVENTIONS §6.1).
  2. Highlights list (can be written into contributions).
  3. Issues/anomalies list (with troubleshooting suggestions).
  4. To-be-added experiments list (return to m05 to supplement the design).
  5. Recommended figure/table list (handed to m09 for planning, m11 for plotting): method comparison bar chart (with 95% CI), box + scatter distributed by seed, learning/scaling curves (with CI bands), confusion/correlation heatmaps, andthe three interpretability plots—SHAP beeswarm (global direction) + bar (importance ranking) + waterfall (single-sample decomposition), produced directly by scripts/explain_shap.py as dpi300 PDF/SVG/PNG.

Ready-to-use scripts (scripts/, all self-tested and passing in python, reusing code_assets' verified statistics)

  • scripts/analyze_results.py: one-click analysis of results-table csv. EDA summary (n/mean±std/median/95% CI/normality) + automatic test selection based on normality and number of groupsauto-selects the test(2 groups normal→Welch t / non-normal→Mann-Whitney; ≥3 groups normal→ANOVA+Tukey / non-normal→Kruskal-Wallis) + pairwise Cohen's d (Hedges correction) + BH-FDR correction across comparisons, outputs summary.json + summary.md. Usage python scripts/analyze_results.py results.csv --group method --metric acc f1；When methods share seeds/folds add --paired-by seed to enable paired t / Wilcoxon signed-rank(higher power, d_z + bootstrap difference CI, uses only shared units, automatically skips when shared<2); run the synthetic demo with no arguments.
  • scripts/significance_test.py: "p + Cohen's d + CI + FDR" function library (welch_t/cohens_d/mean_diff_ci/bootstrap_ci/benjamini_hochberg/compare_two）。__main__ each function aligned with scipy/statsmodels and printing ALL PASS. Reuse ../../../code_assets/stats_tests.py。
  • scripts/make_figs.py: publication-level matplotlib template (OO interface, constrained_layout, colorblind-friendly viridis, error bars, dpi300 vector PDF/SVG/PNG). builder:grouped_bar_ci/box_strip/line_with_band/heatmap + save_all。python scripts/make_figs.py produces the four-panel demo figure.
  • scripts/leakage_overfit_check.py: Pure numpy/pandas train/val/test gap (overfitting/drift) + feature-label high-correlation leakage + train/test duplicate rows + near-constant column warnings; automatically degrades when deepchecks is missing (no hard dependency).python scripts/leakage_overfit_check.py Run a synthetic demo with "implanted leakage".
  • scripts/explain_shap.py: SHAP model interpretability plotting. Dispatch TreeExplainer(tree model fast path) → shap.Explainer(unified entry point automatically picks Tree/Linear) → KernelExplainer(model-agnostic fallback, background set sampled with shap.kmeans/sample for about 100 rows to control cost); encapsulates SHAP plotting pitfalls (each plot show=False → plt.gcf() → reuse make_figs save_all to save dpi300 PDF/SVG/PNG), producing three plots: beeswarm (global direction) + bar (importance ranking) + waterfall (single-sample decomposition). Gracefully degrades and skips when shap is not installed, exit 0 (mimicking deepchecks handling, no hard dependency).python scripts/explain_shap.py Run make_classification+RandomForest synthetic self-test. Pitfalls: strong correlation dilutes contribution, SHAP is not causal, KernelExplainer is expensive.
  • examples/worked_example.py: End-to-end EDA→significance→plots→leakage checkup→filled-in example_report.md, all written to examples/example_out/。
  • assets/result_analysis_report_template.md: Four-part (phenomenon→cause→evidence→significance to the paper) report template, including highlights/anomalies/list of experiments to be added.

Connection

Highlights → m07 writing support; anomalies/deficiencies → go back to m05 to add experiments or go back to m03 to propose new ideas; conclusions written to db09. Honestly label verified/unverified (CONVENTIONS §4).

Per-tool notes on real tool endpoints/APIs/parameters and known pitfalls are in references.md。