SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsmodel-comparison

model-comparison

Agents & models

View repository

Updated 10-15 19:55

About this skill

Multi-model performance comparison (portable)

Based onthe same data split, run multiple algorithms side by side, evaluate uniformly and generate anobjective fact report。

Design principles (important changes in v3)：

  • State facts only, make no subjective recommendations: Removed the old version's "five-dimension weighted composite score" and "gatekeeping elimination". Subjective weights essentially compress multiple incomparable metrics into a single number, losing information and introducing arbitrary assumptions; gatekeeping thresholds vary enormously across businesses.
  • Pareto frontier (no subjective weights): On the five clearly defined objectives of OOT AUC / OOT KS / BCR@10% / KS Gap / PSI, identify the candidate set that is not strictly dominated by any algorithm.
  • LLM reasoning takes over: The end of the report provides a "list of known facts + open questions", handed to the AI in the conversation to reason about trade-offs in light of the business context.
  • Same data, same split → eliminates interference from data differences on comparison conclusions
  • Raw statement of multi-dimensional metrics → OOT AUC/KS/BCR/Brier + KS Gap + PSI + feature count + interpretability
  • Statistical significance → DeLong test answers "is the gap real or sampling noise"

Recommended workflow: tune first, then compare

The question "which algorithm is better" is only fair when compared afterall three algorithms have been tuned: Comparing under default hyperparameters can only serve as an initial screening. The recommended three-step call is:

1. xgb-tuning  → 产出 xgb_tuning_best_*_meta.json
2. lr-tuning   → 产出 lr_tuning_best_*_meta.json
3. dnn-tuning  → 产出 dnn_tuning_best_*_meta.json
4. model-comparison --use_tuned   → 自动扫描上述产物使用最优超参跑对比

Parameter description

Parameter Required Default value Description
--data_path / -d ✅ - Data file path (parquet/csv)
--target / -t ✅ - Target variable column name (0/1 binary classification)
--time_col busi_dt Time column (used for OOT split and monthly performance)
--train_filter Automatic split Training set filter condition
--oot_filter Split out by time OOT test set condition
--oot_ratio 0.20 OOT proportion
--val_ratio 0.25 Val proportion
--random_seed 42 Random seed
--exclude_cols - Excluded columns, comma-separated
--features - Specified features, comma-separated; if not passed, inferred automatically
--algorithms xgb,lr,dnn List of algorithms to compare, comma-separated
--scenario general general / scorecard / fraud / stability_first
--use_tuned flag Off Automatically scan the models directory to load the latest tuned optimal parameters
--tuned_params_file - Explicitly specify {"xgb":{},"lr":{},"dnn":{}} JSON
--config - JSON that manually overrides each algorithm's hyperparameters (takes priority over tuned)
--model_name Automatically generated Artifact name prefix
--output_dir ./outputs/<ts> Artifact output directory

Scenario presets (--scenario）

The scenario serves only as abusiness context label, for use in LLM reasoning; it no longer participates in weighted scoring and does not perform gatekeeping elimination.

Scenario Description
general General purpose; first comparison exploration
scorecard Scorecard / white-box compliance; LR + WoE naturally advantageous
fraud Anti-fraud / high risk; focus on OOT KS / BCR
stability_first Existing model operations; focus on PSI and monthly KS

Execution method

Default three algorithms + general scenarios:

python scripts/comparison.py \
  --data_path ./data.parquet --target y_label \
  --exclude_cols "cust_code,busi_dt" \
  --output_dir ./outputs/compare

Scorecard scenarios (only run XGB + LR):

python scripts/comparison.py \
  --data_path ./data.parquet --target y_label \
  --algorithms "xgb,lr" --scenario scorecard \
  --output_dir ./outputs/compare

Use historical tuning optimal parameters:

python scripts/comparison.py \
  --data_path ./data.parquet --target y_label \
  --use_tuned --scenario fraud \
  --output_dir ./outputs/compare

Comparison dimensions (raw metrics, no weighting)

Dimension Field Description
Effectiveness OOT AUC / KS / Gini / BCR@10% / Precision@10% / Recall@10% / Brier / LogLoss Full set of raw metrics displayed
Generalization KS Gap = Train KS - OOT KS Smaller is better; >0.10 flags overfitting risk
Stability PSI (OOT vs Train score distribution) + monthly KS fluctuation σ Smaller is more stable; ≥0.25 flags significant drift
Interpretability LR strong / XGB medium / DNN weak Text label only, not involved in computation
Conciseness Number of features entered into the model For reference only

Statistical significance (DeLong test)

Pairwise algorithm AUC differences are tested for asymptotic normality via Fast-DeLong (O(N log N), completes in seconds even with 100k+ samples):

  • p < 0.05 → difference is statistically significant
  • p ≥ 0.05 → difference may be sampling fluctuation

Outputs

  1. Comparison report(Markdown, inverted pyramid structure):

    • Executive summary (objective metrics overview + Pareto frontier markers,no recommendation column）
    • Multi-algorithm overlay curves (ROC/KS/Lift)
    • Data overview
    • Detailed metrics summary table (train/val/oot × 8 metrics)
    • Pareto frontier candidate set(no subjective weighting)
    • DeLong significance test
    • Stability (real PSI + monthly KS)
    • Sample-level consistency (Top-K Jaccard / Spearman / disagreement count)
    • Cross-algorithm feature overlap (Jaccard + consensus features)
    • Independent evaluation plots per algorithm
    • LLM reasoning guidance: fact list + open questions, handed over to the AI in the conversation for reasoning and trade-offs
    • Background knowledge on applicable scenarios for each algorithm

  2. Model files for each algorithm: XGB(.json) / LR(.joblib) / DNN(.pt, including imputer+scaler metadata)

Relationship with other skills

Skill Responsibility Relationship
xgb-tuning / lr-tuning / dnn-tuning Hyperparameter tuning optimization Prerequisite: tuning is required before fair comparison
xgb-modeling / lr-modeling / dnn-modeling Baseline modeling Reused to train comparison models
model-explanation Model interpretation Follow-up: interpret the best model after comparison is complete

Notes

  1. Fair comparison: "which algorithm is better" is only fair when all three algorithms have been tuned before comparison
  2. Same data, same split: this skill ensures all algorithms use exactly the same data split
  3. DeLong test: only p < 0.05 is considered a statistically significant difference
  4. Pareto frontier: only identifies the candidate set not strictly dominated by any algorithm, without subjective ranking
  5. Scenario label：--scenario is used only as a text label and does not participate in weighted scoring
  6. Artifact location: Models and reports are saved to <output_dir>/