xgb-tuning

XGBoost Parameter Tuning (portable)

The only entry pointfor XGBoost tuning, based on _vendor/tuning_engine.TuningEngine. Core design:

1. Intelligent baseline parameter inference — recommends a reasonable starting point based on data characteristics
2. Model state diagnosis — overfitting/underfitting determination (diagnose_model）
3. Constrained Bayesian search — diagnosis conclusions directionally shrink the Optuna search space
4. User knowledge fusion — accepts adjustments to the strategy based on the user's domain experience

---

Tuning process

用户需求 → 数据特征分析 → LLM 推断基线参数 → 训练评估 → 诊断分析 → 参数调整 → 迭代直到满意
                                         ↑                                        ↓
                                         └───────────── 用户反馈/知识输入 ─────────────┘

---

Execution modes

Mode	Trigger condition	Behavior
Interactive(default)	User says "tune parameters"/"help me tune it"/"optimize it"	Pause after each round to wait for user feedback
AUTO	User says "automatic tuning"/"tune it to optimal for me"/"keep tuning until convergence"	The Agent iterates automatically until convergence, outputting progress each round

Default mode: Interactive (safer, user-controllable)

Interactive mode behavior specification

1. Must pause after a single round of tuning, output a structured diagnostic report, and wait for user feedback
2. Possible user feedback:
   • "Continue" / "tune another round" → execute the next round
   • "The gap is still large" / "be more conservative" → adjust the strategy and then execute
   • "That's enough" / "stop" → generate the final report
3. Prohibited: continuously executing multiple rounds of tuning in interactive mode

AUTO mode behavior specification

1. After each round of tuning, also output a complete structured diagnostic report (same format as interactive mode), then automatically proceed to the next round
2. Convergence conditions: Gap < 0.03 or improvement < 0.002 for 2 consecutive rounds
3. Automatically generate the final report after convergence

---

Parameter description

The general parameter spec is defined in _vendor/xgb_cli.py（domain=tuning）。

Parameter	Required	Default value	Description
--data_path / -d	✅	-	Data file path (parquet/csv)
--target / -t	✅	-	Target variable column name (0/1 binary classification)
--features / -f	✅	-	Feature list, comma-separated
--time_col		busi_dt	Time column name
--train_filter		Automatic split	Training set filter condition (pandas query)
--val_filter		val_ratio split out	Validation set filter condition (has fully replaced the old --test_filter）
--oot_filter		Split by time	OOT test set condition
--oot_ratio / --val_ratio		0.20 / 0.25	Automatic split ratio
--random_seed		42	Random seed
--exclude_cols		-	Excluded columns, comma-separated
--params / -p		Default parameters	Current parameters (JSON;recommended to put in --config 's params field）
--baseline / -b		-	Baseline parameters (JSON;recommended to put in --config 's baseline field）
--round / -r		0	Current round
--prev_val_metric		-	Previous round val metric (used for convergence judgment)
--max_rounds		5	Maximum tuning rounds
--auto		-	Enable auto-tuning mode (flag)
--metric		ks	Evaluation metric (auc/ks）
--model_name		Auto-generated	Model name (without extension)
--report_output / -o		Auto-generated	Report output path
--output_dir		./outputs/<ts>	portable only: artifact output directory
--warm_start		-	WarmStartBundle JSON string or file path
--config		-	JSON configuration file path (by --config automatically injected, generally no need to pass manually)

--params / --baseline When passing complex JSON, prefer putting it in --config, to avoid command-line double-quote escaping issues.

---

Intelligent inference of baseline parameters

The Agent should, based on thedata summaryoutput by tuner.py, infer reasonable baseline parameters rather than using fixed default values.

Data summary fields

tuner.py will output the following data features for the Agent to analyze:

Field	Description	Affected parameters
train_samples	Training set sample size	max_depth, n_estimators
oot_samples	OOT sample size	subsample
n_features	Number of features	colsample_bytree
pos_rate	Positive sample rate	min_child_weight, scale_pos_weight

Inference rules

Sample size and tree depth

Training set sample size	max_depth suggestion	Reason
< 50k	3	Few samples, low complexity to prevent overfitting
50k - 200k	4	Medium sample size, moderate complexity
200k - 1M	5	Sufficient samples, can be slightly more complex
> 1M	5-6	Large samples support higher complexity

Positive sample rate and leaf nodes

Positive sample rate	min_child_weight suggestion	Reason
< 1%	300+	Very few positive samples, larger leaf nodes needed to prevent choking
1% - 5%	100-200	Imbalanced, appropriate constraints
5% - 20%	50-100	Relatively balanced, standard constraints
> 20%	20-50	Balanced data, can be slightly looser

Number of features and sampling rate

Number of features	colsample_bytree suggestion	Reason
< 20	0.9-1.0	Few features, make full use of them
20 - 50	0.7-0.9	Medium features, moderate sampling
> 50	0.5-0.7	Many features, increase randomness

Inference example

数据摘要:
  训练集: 150,000 样本
  OOT: 50,000 样本
  特征数: 35 个
  正样本率: 2.5%

Agent 推断基线参数:
  max_depth: 4        <- 样本量中等
  min_child_weight: 150  <- 正样本率低
  colsample_bytree: 0.8  <- 特征数中等
  reg_alpha: 0.3      <- 特征多，适当正则
  reg_lambda: 1.0
  learning_rate: 0.05
  n_estimators: 500
  subsample: 0.8

---

Scenario-based strategy

The Agent should, based on thescenario informationprovided by the user, adjust the tuning strategy.

Financial risk control scenario

Features: Long-term model use, stability first

Parameter	Suggested value	Reason
max_depth	3-4	Low complexity, resistant to overfitting
min_child_weight	150+	Leaf nodes must be stable
reg_alpha	0.3-0.5	Strong regularization
reg_lambda	1.0-2.0	Strong regularization

Tuning priority: Gap control > KS improvement

Termination condition: Gap < 0.02, acceptable even if KS is slightly lower

Marketing response scenario

Features: Short-term use, effectiveness first

Parameter	Suggested value	Reason
max_depth	4-5	Allow higher complexity
min_child_weight	50-100	Slightly looser
reg_alpha	0.1-0.2	Moderate regularization

Tuning priority: KS improvement > Gap control

Termination condition: KS meets target, Gap < 0.05 acceptable

Balanced scenario (default)

Features: Balances performance and stability

Parameter	Recommended value
max_depth	4-5
min_child_weight	100
reg_alpha	0.1-0.3
reg_lambda	0.5-1.0

Termination condition: KS >= 0.30 and Gap < 0.03

---

Execution method

Complex parameters (params/baseline) are recommended to be passed via --config JSON file:

# 自动调优（推荐：通过 config.json 传复杂参数）
python scripts/tuner.py \
  --data_path ./data.parquet --target y_label --features "f1,f2,f3" \
  --auto --max_rounds 5 --output_dir ./outputs/tuning \
  --config ./config.json

config.json Example:

{
  "params": {"max_depth": 4, "learning_rate": 0.05, "n_estimators": 500},
  "baseline": {"max_depth": 4, "learning_rate": 0.1}
}

Interactive mode (single-round tuning)

python scripts/tuner.py \
  --data_path ./data.parquet --target y_label --features "f1,f2,f3" \
  --round 1 --output_dir ./outputs/tuning

AUTO mode (automatic tuning loop)

python scripts/tuner.py \
  --data_path ./data.parquet --target y_label --features "f1,f2,f3" \
  --auto --max_rounds 5 --metric auc \
  --output_dir ./outputs/tuning

The script outputs model, report, and state updates through the single exit protocol [RESULT:{json}] ;The LLM should not restate charts already produced by the script。

---

Diagnostic knowledge base

Model state diagnosis

Diagnostic result	Judgment condition	Description
Overfitting	Train-OOT Gap > 0.05	Training set performance far exceeds test set; the model memorizes training data
Slight overfitting	Gap ∈ [0.04, 0.05]	Some overfitting risk exists; needs attention
Good fit	Gap ∈ [0.02, 0.04]	Model generalization ability is normal
Underfitting	OOT AUC < 0.55 and Gap < 0.02	Model fitting ability is insufficient
Converged	Improvement < 0.001 for 2 consecutive rounds	Limited room for optimization; can stop

Overfitting signals

• Train AUC keeps rising, OOT AUC declines or stagnates
• Train-OOT Gap increases round by round
• Validation set performance is unstable

Underfitting signals

• Both Train AUC and OOT AUC are low
• Performance keeps improving after increasing training rounds
• Gap is very small but overall AUC is insufficient

---

XGBoost parameter semantics

Parameter	Effect	Value range	When overfitting	When underfitting
max_depth	Tree depth, controls model complexity	2-8	↓ Decrease	↑ Increase
min_child_weight	Minimum sample weight per leaf node	10-300	↑ Increase	↓ Decrease
reg_alpha	L1 regularization strength	0-2.0	↑ Increase	↓ Decrease
reg_lambda	L2 regularization strength	0.1-10	↑ Increase	↓ Decrease
subsample	Sample sampling rate	0.5-1.0	↓ Decrease	↑ Increase
colsample_bytree	Feature sampling rate	0.5-1.0	↓ Decrease	↑ Increase
learning_rate	Learning rate	0.005-0.15	↓ Decrease	↑ Increase
n_estimators	Number of trees	100-1000	↓ Decrease	↑ Increase

Parameter tuning priority

Overfitting scenario(by priority):

1. Increase reg_alpha / reg_lambda(most direct)
2. Decrease max_depth(control complexity)
3. Increase min_child_weight(Restrict splits)
4. Decrease subsample / colsample_bytree(increase randomness)

Underfitting scenarios(in priority order):

1. Increase max_depth(increase complexity)
2. Increase n_estimators(more iterations)
3. Decrease regularization parameter
4. Appropriately increase learning_rate

---

User instruction understanding

User expression	Parameter mapping	Adjustment magnitude
"a bit more regularization"	reg_alpha ↑ or reg_lambda ↑	+50%~100%
"a bit less regularization"	reg_alpha ↓ or reg_lambda ↓	-30%~50%
"deeper tree depth"	max_depth ↑	+1
"shallower tree depth"	max_depth ↓	-1
"lower learning rate"	learning_rate ↓	-30%~50%
"higher learning rate"	learning_rate ↑	+30%~50%
"train more rounds"	n_estimators ↑	+50%~100%
"train fewer rounds"	n_estimators ↓	-30%~50%
"prevent overfitting"	Combined: regularization↑, depth↓, subsample↓	Combined adjustment
"fit a bit stronger"	Combined: depth↑, regularization↓	Combined adjustment
"a bit more aggressive"	learning_rate ↑, max_depth ↑	Larger magnitude
"a bit more conservative"	learning_rate ↓, regularization↑	Smaller magnitude
"continue automatic tuning"	Start a new AUTO round from the current parameters	-
"just use this" / "confirm"	End tuning and output the final configuration	-

---

Tuning strategies

Strategy 1: Anti-overfitting

Applicable conditions：Gap > 0.05

Adjustment directions：

• reg_alpha: current value × 2 (e.g., 0.1 → 0.2)
• reg_lambda: current value × 1.5
• max_depth: current value - 1 (minimum 2)
• min_child_weight: current value × 1.5

Strategy 2: Enhanced fitting

Applicable conditions: OOT AUC < 0.58 and Gap < 0.03

Adjustment directions：

• max_depth: current value + 1 (maximum 8)
• n_estimators: current value × 1.5
• reg_alpha: current value × 0.5
• learning_rate: current value × 1.2

Strategy 3: Fine-tuning

Applicable conditions: Gap ∈ [0.03, 0.05], model state is good

Adjustment directions：

• learning_rate: small adjustment ±20%
• subsample: small adjustment ±10%
• Other parameters remain unchanged

Strategy 4: Convergence determination

Condition: OOT metric improvement < 0.001 for 2 consecutive rounds

Behavior: stop tuning and output the final result

Strategy 5: Directed search under constrained space

In AUTO mode, tuner.py calls TuningEngine.run_round(diagnosis, tried_directions) each round to run 5 Optuna trials within the diagnostic constraint space, directly taking the optimal parameters of this round into the next round.tried_directions automatically records the increase/decrease direction and effect of parameters each round; if a certain direction does not improve, that dimension will be automatically frozen in the next round. The Agent does not need manual tracking, but in each round's report it should state "this round's diagnosis is XX → the focus of the search space is XX" to help users understand the tuning reasoning.

---

Output format specification

Core principle: each round must output completely Do not output only the final tuning report. After each round of tuning is completed, whether in interactive or AUTO mode,you mustimmediately output the complete diagnostic analysis process and results for that round, including: parameter changes and reasons for adjustment, details of training metrics, comparison with the previous round, diagnostic conclusions, and next-step recommendations. Users need to see the diagnostic reasoning process for each round, not just the final parameters.

Single-round tuning output (must be used for each round, applicable to both interactive and AUTO modes)

After each round of tuning is completed,you mustoutput the following structured information:

### 第 N 轮调优结果

**参数变化**:
| 参数 | 上一轮 | 本轮 | 调整原因 |
|------|-------|------|----------|
| max_depth | 4 | 3 | 降低过拟合 |
| reg_alpha | 0.1 | 0.3 | 增强正则化 |

**效果对比**:
| 指标 | 上一轮 | 本轮 | 变化 |
|------|-------|------|------|
| OOT KS | 0.17 | 0.18 | +0.01 ✓ |
| OOT AUC | 0.72 | 0.73 | +0.01 ✓ |
| Gap (KS) | 0.06 | 0.04 | -0.02 ✓ |

**诊断结论**: 轻微过拟合（Gap 下降但仍 > 0.03）

**下一步建议**: 可继续微调正则化，或接受当前结果

Final report (generated at the end of tuning, cannot replace round-by-round output)

When the user confirms the end or AUTO mode converges, after the round-by-round output is completed, additionally generate a complete summary report:

Note: The final report is a summary supplement to the round-by-round output and cannot replace the round-by-round output. Even in AUTO mode, you must first output round by round and then summarize.

# XGBoost 调参报告

## 1. 调优概览

| 项目 | 内容 |
|------|------|
| 执行模式 | 交互式 / AUTO |
| 总轮数 | 3 |
| 收敛原因 | Gap < 0.03 达标 / 用户确认停止 |

## 2. 调参推演记录

| 轮次 | 参数 (depth/eta/reg) | OOT KS | OOT AUC | Gap (KS) | 诊断 | 调整决策 |
|------|---------------------|--------|---------|----------|------|----------|
| 基线 | 4 / 0.1 / 0.1 | 0.16 | 0.71 | 0.08 | 过拟合 | 降低 depth |
| R1 | 3 / 0.1 / 0.2 | 0.17 | 0.72 | 0.05 | 轻微过拟合 | 增强正则化 |
| R2 | 3 / 0.08 / 0.5 | 0.18 | 0.73 | 0.03 | 良好 | 收敛停止 |

## 3. 最终效果

| 指标 | 基线 | 最终 | 提升 |
|------|------|------|------|
| OOT KS | 0.16 | 0.18 | +0.02 |
| OOT AUC | 0.71 | 0.73 | +0.02 |
| Gap (KS) | 0.08 | 0.03 | -0.05 |

## 4. 最终参数

```json
{
  "max_depth": 3,
  "learning_rate": 0.08,
  "reg_alpha": 0.5,
  "reg_lambda": 1.0,
  "min_child_weight": 100,
  "subsample": 0.8,
  "colsample_bytree": 0.8,
  "n_estimators": 500
}

5. Tuning conclusion

Compared with the baseline model, the final model:

• OOT KS improved by 0.02 (0.16 → 0.18)
• OOT AUC improved by 0.02 (0.71 → 0.73)
• Gap (KS) decreased by 0.05 (0.08 → 0.03)
• Stability improved significantly and can be safely deployed

If further exploration is needed, please provide your tuning recommendations.

---

## 与其他技能的关系

| 技能 | 职责 | 关系 |
|------|------|------|
| `xgb-modeling` | 基线建模 | 前置：需先用其训练出基线模型 |
| `model-explanation` | SHAP 解释 | 后续：调参完成后解释最优模型 |
| `model-comparison` | 多算法对比 | 平行：可与 LR/DNN 调参后做公平对比 |
| `auto-experiment` | 特征探索 | 区别：本 Skill 调参数，auto-experiment 探索特征 |

---

## 注意事项

1. **数据要求**：目标变量必须为 0/1 二分类
2. **特征要求**：需提供已筛选的特征列表（`--features` 必填）
3. **基线参数**：可传入自定义基线参数，否则使用默认值
4. **收敛判定**：连续2轮提升不足 0.001 自动停止
5. **最大轮数**：默认最多 5 轮，避免过度调优
6. **复杂 JSON**：`--params` / `--baseline` 等复杂 JSON 优先通过 `--config config.json` 传入
7. **产物位置**：模型和报告保存到 `<output_dir>/models/` 和 `<output_dir>/`