Based on scripts/analyzer.py main script, perform univariate-level distribution analysis or predictive power evaluation on specified features.
Functional positioning
| Mode | Target variable | Core function | Use case |
|---|---|---|---|
| Data exploration mode | ❌ None | Distribution analysis, cross distribution | Just uploaded data, initially understand feature distribution |
| Feature selection mode | ✅ Yes | IV value, binning table, selection suggestions | Quickly filter features before modeling |
Parameter description
| Parameter | Required | Default value | Description |
|---|---|---|---|
--data_path |
✅ | - | Data file path (parquet/csv) |
--features |
✅ | - | Feature column names to analyze, comma-separated, or "all" |
--target |
- | Target variable column name (optional; if present, enter selection mode) | |
--exclude_cols |
- | Columns to exclude, comma-separated | |
--binning_method |
quantile |
Binning method:quantile(equal frequency) / distance(equal width) |
|
--n_bins |
10 |
Number of bins | |
--cross |
- | Two features for cross distribution, comma-separated | |
--output |
Automatically generated | Report output name | |
--output_dir |
./outputs/<ts> |
Artifact output directory | |
--config |
- | JSON configuration file path |
Execution method
Data exploration mode (no target variable)
python scripts/analyzer.py \
--data_path ./data.parquet --features "age,income,score" \
--output_dir ./outputs/uni_run
Feature selection mode (with target variable)
python scripts/analyzer.py \
--data_path ./data.parquet --features "age,income,score" --target y_label \
--output_dir ./outputs/uni_run
Cross distribution analysis
python scripts/analyzer.py \
--data_path ./data.parquet --features "age,income" --cross "age,income" \
--output_dir ./outputs/uni_run
Common scenarios
Scenario 1: Data exploration (no target variable)
View single-feature distribution:
python scripts/analyzer.py --data_path ./data.parquet --features "age,income,score" --output_dir ./outputs/uni_run
Specify equal-width binning:
python scripts/analyzer.py --data_path ./data.parquet --features "age" --binning_method distance --n_bins 5 --output_dir ./outputs/uni_run
Scenario 2: Cross distribution analysis
View the joint distribution of two features:
python scripts/analyzer.py --data_path ./data.parquet --features "age,income" --cross "age,income" --output_dir ./outputs/uni_run
Scenario 3: Feature selection (with target variable)
Calculate IV value and evaluate feature predictive power:
python scripts/analyzer.py --data_path ./data.parquet --features "age,income,score" --target y_label --output_dir ./outputs/uni_run
Report output structure
Data exploration mode (no target variable)
| Section | Content |
|---|---|
| 1. Data overview | Sample size, number of analyzed features |
| 2. Basic statistics | Mean, median, missing rate, cardinality, etc. of each feature |
| 3. Distribution analysis | Binning distribution table for each feature (interval, sample count, proportion) + bar distribution chart |
| 4. Cross distribution | Joint distribution table of two features (optional) |
| 5. Data quality flags | Flags for issues such as excessively high missing rate and low cardinality |
Feature selection mode (with target variable)
| Section | Content |
|---|---|
| 1. Data overview | Sample size, positive sample rate, number of analyzed features |
| 2. Basic statistics | Mean, median, missing rate, cardinality, etc. of each feature |
| 3. IV value analysis | Ranking of IV values for each feature and binning detail table |
| 4. Data Quality Flags | Flag issues such as excessively high missing rate and low cardinality |
| 5. Screening Recommendations | List of features recommended to keep/remove |
Binning Description
Equal-frequency binning (quantile)
Divide by data quantiles, with roughly equal sample size in each bin. Suitable for unevenly distributed data.
Equal-width binning (distance)
Divide into equal intervals by numeric range. Suitable for evenly distributed data.
Special Value Handling
- Missing values: in a separate bin, labeled as
Missing - Zero values: in a separate bin when the proportion of zero values is > 5%
- Negative numbers: handle separately when negative numbers exist
Output Examples
Distribution Analysis Table
特征: age | 分箱方式: 等频 | 分箱数: 10
| 区间 | 样本数 | 占比 | 累计占比 |
|---------------|---------|---------|----------|
| Missing | 156 | 1.56% | 1.56% |
| [18, 23) | 984 | 9.84% | 11.40% |
| [23, 28) | 1,012 | 10.12% | 21.52% |
| ... | ... | ... | ... |
Cross Distribution Table
age × income 交叉分布
| | 低收入 | 中收入 | 高收入 | 合计 |
|--------------|----------|----------|----------|---------|
| [18, 25) | 800 | 300 | 134 | 1,234 |
| [25, 35) | 500 | 1,200 | 756 | 2,456 |
| [35, 45) | 200 | 800 | 1,000 | 2,000 |
| ... | ... | ... | ... | ... |
IV Binning Detail Table (with target variable)
特征: age | IV = 0.1523
| 区间 | 样本数 | 正样本数 | 正样本率 | WoE | IV |
|---------------|---------|----------|----------|---------|---------|
| [18, 23) | 984 | 156 | 15.85% | 0.32 | 0.0234 |
| [23, 28) | 1,012 | 98 | 9.68% | -0.15 | 0.0089 |
| ... | ... | ... | ... | ... | ... |
Relationship with feature-analysis
| Dimension | univariate-analysis | feature-analysis |
|---|---|---|
| Positioning | Single-feature quick analysis | Full-feature in-depth analysis |
| Target variable | Optional | Required |
| Correlation analysis | ❌ | ✅ |
| PSI stability | ❌ | ✅ |
| Solution recommendations | ❌ | ✅ Four solutions |
| Typical usage | Quick screening before modeling | Complete feature engineering report |
Recommended workflow：univariate-analysis(quick screening) → feature-analysis(in-depth analysis)
Notes
- Number of features: it is recommended to analyze no more than 50 features at a time; process large numbers of features in batches
- Number of bins: 10 bins by default, adjustable according to data volume (5 bins recommended when data volume is small)
- Cross distribution: only supports crossing of two features; it is recommended to select discrete or already binned features
- IV calculation: requires the target variable to be 0/1 binary classification
- Artifact location: the report is saved to
<output_dir>/