feature-analysis

Modeling Feature Analysis Report (portable)

Based on scripts/analyzer.py main script, performs a comprehensive analysis of dataset features and generates a Markdown report.

---

Parameter Description

Parameter	Required	Default Value	Description
--data_path	✅	-	Data file path (parquet/csv)
--target	✅	-	Target variable column name (0/1 binary classification)
--exclude_cols		-	Excluded columns, comma-separated
--baseline_filter		-	PSI baseline data condition (pandas query)
--comparison_filter		-	PSI comparison data condition (pandas query)
--top_n		10	Output binning details for the top N features by IV rank (default 10)
--specified_features		-	Specify feature binning details, comma-separated
--output		feature_analysis_report.md	Report output name
--output_dir		./outputs/<ts>	Artifact output directory
--config		-	JSON configuration file path

---

Execution Method

python scripts/analyzer.py \
  --data_path ./data.parquet --target y_label \
  --exclude_cols "cust_code,busi_dt" \
  --baseline_filter "busi_dt <= '20250501'" \
  --comparison_filter "busi_dt > '20250701'" \
  --output_dir ./outputs/fa_run

---

Common Scenarios

Scenario 1: Basic Analysis (excluding PSI)

Suitable for datasets without a time dimension:

python scripts/analyzer.py --data_path ./data.parquet --target y_label --output_dir ./outputs/fa_run

Scenario 2: Full Analysis (including PSI)

Suitable for datasets with time split conditions:

python scripts/analyzer.py --data_path ./data.parquet --target y_label \
  --baseline_filter "busi_dt <= '20250501'" \
  --comparison_filter "busi_dt > '20250701'" \
  --output_dir ./outputs/fa_run

Scenario 3: Univariate Deep Analysis

Output the complete binning detail table for the IV Top N or specified features:

python scripts/analyzer.py --data_path ./data.parquet --target y_label --top_n 10 --output_dir ./outputs/fa_run
python scripts/analyzer.py --data_path ./data.parquet --target y_label --specified_features "umeng_ALL,bscore" --output_dir ./outputs/fa_run

---

Report Output Structure

The generated feature_analysis_report.md contains the following sections:

Section	Content
1. Data Overview	Sample size, number of features, positive sample rate
2. Basic Statistical Analysis	Mean, standard deviation, missing rate, skewness, kurtosis, etc. of each feature
3. IV Value Analysis	IV Top 20 ranking, IV distribution statistics
4. PSI Stability Analysis	PSI ranking, list of unstable features
5. Correlation Analysis	Highly correlated feature pairs, collinearity handling suggestions
6. Comprehensive Recommendations	Recommended features to keep/remove
7. Modeling Feature Plans	Four feature selection plans (full/de-collinearized/high-IV/stability-first)
8. Univariate Deep Analysis	Outputs binning details for the Top 10 features by default, adjustable via --top_n adjustment

---

Feature Selection Plan Description

The report automatically generates four modeling feature plans:

Plan	Selection Criteria	Applicable Scenario
Plan 1: Full Inclusion	Basic qualified pool (IV>=0.02, PSI<0.25, missing rate<50%)	Tree models such as XGBoost/LightGBM
Plan 2: De-collinearity Standard	Basic pool + greedy removal of |r|>=0.7	Logistic regression, scorecards
Plan 3: High Predictive Power Selection	IV>=0.1 + de-collinearity	Limited features, high interpretability requirements
Solution 4: Stability first	PSI<0.1 + decorrelation	Online deployment, high stability requirements

---

Relationship with other Skills

Skill	Purpose differences
feature-analysis	Full-feature deep analysis + solution recommendation, requires target variable
data-profiling	Quickly understand the data profile, no deep analysis
univariate-analysis	Rapid analysis of a small number of features, excluding full correlation and solution recommendation

Suggested workflow：data-profiling(quick scan) → feature-analysis(deep analysis + solution recommendation) → xgb-modeling(modeling)

---

Notes

1. Target variable: must be numeric
2. PSI analysis: must provide both baseline_filter and comparison_filter, otherwise skip
3. Correlation: only valid for numeric features, non-numeric columns are automatically skipped
4. Large datasets: the report displays IV Top 20 by default, complete data is in the file
5. Artifact location: the report is saved to <output_dir>/