auto-experiment

Autonomous experiment loop (portable)

Automatically discover feature groupings in the data, execute a four-phase progressive exploration experiment, and fully display the exploration logic and quantitative results in each round.

Core Concept

Drawing on the autonomous loop idea of autoresearch, upgraded togroup-level exploration：

发现特征组 → 组独立评估 → 组间叠加 → 组级消融 → 精细筛选

Four-Phase Exploration Flow

Phase 1: Independent Evaluation of Feature Groups

Automatically discover feature groupings in the data (aggregate by prefix: firefly_*, mob3_*, umeng_* , etc.)
Model each group separately and evaluate each group's independent predictive power
Output the independent AUC/KS ranking of each group

Phase 2: Incremental Stacking Between Groups

Start from the group with the best performance in Phase 1
Gradually stack the next strongest group and observe the marginal increment
Find the optimal combination point

Phase 3: Group-Level Ablation Analysis

Starting from the full feature set, remove one group at a time
Quantify the irreplaceability of each group
If the metric does not decrease after removal, it means that group's information has already been covered by other groups

Phase 4: Fine-Grained Screening Within Groups

Within the optimal combination, try removing features one by one
Eliminate redundant features and streamline the model

Parameter Description

Parameter	Required	Default Value	Description
`--data_path`	✅	-	Data file path (parquet/csv)
`--target`	✅	-	Target variable column name (0/1 binary classification)
`--exploration`	✅	-	Exploration direction description, such as "the contribution of different feature groups to mob3 delinquency"
`--max_rounds`		`5`	Maximum number of experiment rounds
`--metric`		`ks`	Optimization metric (auc/ks)
`--direction`		`maximize`	Optimization direction (maximize/minimize)
`--significance`		`2.0`	Significance threshold (MAD multiple)
`--baseline_features`		-	Baseline feature list, JSON format (uses all features by default)
`--time_col`		`busi_dt`	Time column name
`--train_filter`		-	Training set filter condition
`--val_filter`		-	Validation set filter condition
`--test_filter`		-	[Deprecated] Equivalent to `--val_filter`, for backward compatibility only
`--oot_filter`		-	OOT test set condition
`--val_ratio`		`0.25`	Proportion of val within train_full
`--oot_ratio`		`0.20`	Proportion of OOT within the full dataset
`--output_dir`		`./outputs/<ts>`	Artifact output directory
`--config`		-	JSON configuration file

Execution Method

python scripts/run_experiment.py \
  --data_path ./data.parquet --target y_label \
  --exploration "不同特征组(firefly/子模型/友盟)对mob3逾期的贡献" \
  --max_rounds 10 \
  --output_dir ./outputs/exp

Common Scenarios

Scenario 1: Explore the contribution of each feature group

python scripts/run_experiment.py \
  --data_path ./data.parquet --target y_label \
  --exploration "不同特征组(firefly模型/子模型/友盟数据)对mob3逾期预测的贡献" \
  --max_rounds 10 --output_dir ./outputs/exp

Scenario 2: Explore features in a specific time window

python scripts/run_experiment.py \
  --data_path ./data.parquet --target y_label \
  --exploration "mob3相关的逾期、还款特征" \
  --max_rounds 8 --output_dir ./outputs/exp

Scenario 3: Explore new groups on an existing baseline

python scripts/run_experiment.py \
  --data_path ./data.parquet --target y_label \
  --baseline_features '["feat1","feat2","feat3"]' \
  --exploration "添加mob6时间窗口特征组" \
  --max_rounds 5 --output_dir ./outputs/exp

Output Content

Core principle: each round must fully output the exploration logic and results Outputting only the final summary is prohibited. After each round of experiments is completed,mustimmediately output: exploration strategy, reasoning logic, hypothesis description, added/removed features, training metric details, comparison with the baseline, significance test, and decision rationale.

Data Overview (before the experiment starts)

Data size, number of features, positive sample rate
Automatically discovered feature groups and their statistics

Single-round output format (must be used in every round)

### Round N/总轮数

**Phase X: 探索策略名称**

**探索逻辑**:
[Phase 1/3/5] 特征组独立评估
目标: 单独测试「firefly」组的 12 个特征
逻辑: 先让每个特征组独立上场，获得各组的独立贡献排名

**本轮假设**: 独立评估特征组「firefly」(12个特征)

**模型评估** (共 12 个特征):
| 指标 | Train | Test | 基线Test | 变化 |
|------|-------|------|----------|------|
| AUC | 0.812 | 0.735 | 0.726 | +0.009 |

**Top-5 特征重要性**:
| # | 特征 | 重要性 |
|---|------|--------|
| 1 | firefly_score [NEW] | 0.2341 |

**显著性检验**: [PASS] 2.3x MAD → 改进显著
**决策**: [+] KEEP - 改进显著

Final summary (output after all rounds)

Experiment overview table (Round/phase/hypothesis/metric/decision)
Final vs baseline comparison
Retained and discarded feature groups/feature list
Next-step recommendations

Statistical significance testing

Use MAD (Median Absolute Deviation) to determine whether an improvement is significant:

Confidence	Mark	Meaning
≥ 2.0×	[PASS]	The improvement may be real
1.0-2.0×	[EDGE]	Above noise but marginal
< 1.0×	[FAIL]	Within the noise range

Relationship with other skills

Skill	Responsibility	Relationship
`xgb-modeling`	Single training and evaluation	Reused by this skill
`xgb-tuning`	Hyperparameter tuning	Can be used after the loop
`feature-analysis`	Feature analysis	Assists hypothesis generation

Notes

Exploration direction: Can be broad (such as "explore the contribution of each feature group"), and the system will automatically discover groups
Round count recommendation: Set to 2-3 times the number of feature groups (for example, 10-15 rounds for 6 groups) to cover multiple exploration phases
Baseline strategy: By default, use all features as the baseline, but a specific feature set can also be specified
Significance threshold: Default 2.0×, and it can be appropriately lowered when the amount of data is small
Interruptible: The user can stop at any time, and existing results will be retained
Artifact location: Reports and models are saved to <output_dir>/