SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsprofiling

profiling

Data & analytics

View repository

Updated 10-15 20:31

About this skill

Profiling Skill

A skill that automatically analyzes the quality and characteristics of a dataset and generates a comprehensive report.

Overview

This skill uses ydata-profiling (formerly pandas-profiling) to perform the following:

  1. ✅ Data quality validation (missing values, duplicates, outliers)
  2. ✅ Statistical analysis (distribution, correlation, skewness/kurtosis)
  3. ✅ Automatic visualization (histograms, correlation heatmaps, etc.)
  4. ✅ HTML report generation and automatic browser opening
  5. ✅ Summary of key findings

Installation

uv package manager (recommended - 10-100x faster)

# uv 설치 (한 번만)
curl -LsSf https://astral.sh/uv/install.sh | sh
# 또는 macOS
brew install uv

# 의존성 설치
cd plugins/data-profiling/skills/profiling
uv pip install -r requirements.txt

Using pip (traditional method)

cd plugins/data-profiling/skills/profiling
pip install -r requirements.txt

Verify installation

python3 -c "import ydata_profiling; print(f'✓ ydata-profiling {ydata_profiling.__version__}')"

Usage

Run the Python script directly

cd plugins/data-profiling/skills/profiling/scripts

# 기본 사용법
python generate_profile.py \
  --data-path "path/to/data.csv" \
  --output-dir "outputs/reports"

# 타겟 컬럼 지정
python generate_profile.py \
  --data-path "samples/datascience/data/raw/creditcard.csv" \
  --target-column "Class" \
  --mode explorative

# 대용량 데이터 샘플링
python generate_profile.py \
  --data-path "large_data.csv" \
  --sample-size 50000 \
  --mode minimal

Run as a Claude Code command

/profile-data --data-path "./data/raw/creditcard.csv" --target-column "Class"

Script file

generate_profile.py

Key Features:

  • Supports various file formats (CSV, Excel, Parquet, JSON, etc.)
  • Automatic sampling (large datasets)
  • Comprehensive analysis via ydata-profiling
  • HTML report generation
  • Automatic browser launch by operating system (macOS, Linux, Windows)
  • Console summary output

Dependencies:

# uv 사용 (권장)
cd plugins/data-profiling/skills/profiling
uv pip install -r requirements.txt

# 또는 pip 사용
pip install -r requirements.txt

Included Packages:

  • pandas - data processing
  • ydata-profiling - automated EDA

Key Parameters:

Parameter Description Default
--data-path Data file path (required)
--target-column Target variable column name None
--sample-size Sampling size None (all)
--mode Profiling mode explorative
--output-dir Report save directory outputs/reports
--no-browser Disable automatic browser opening False

Profiling Mode

1. minimal

  • Execution time: ~1분
  • Contents: basic statistics, missing values, data types
  • Recommended situation: quick data check

2. default

  • Execution time: ~3분
  • Contents: distribution, correlation, basic visualization
  • Recommended situation: general EDA

3. explorative (advanced)

  • Execution time: ~5-10분
  • Contents: all analyses + interaction analysis
  • Recommended situation: in-depth exploration

HTML Report Structure

The generated report includes the following sections:

1. Overview

  • Dataset info (number of rows/columns, memory usage)
  • Variable types (numeric, categorical, datetime, etc.)
  • Warnings (number of warnings)

2. Variables (analysis by variable)

For each variable:

  • Statistics: Mean, Median, Min, Max, Std, Skewness, Kurtosis
  • Distribution visualization: Histogram, KDE plot
  • Missing values: Missing count & percentage
  • Unique values: Distinct count, Most common values
  • Extreme values: Minimum/Maximum values

3. Interactions

  • Scatter plot matrix (scatter plot between variables)
  • Pairwise relationships

4. Correlations

  • Pearson correlation matrix
  • Spearman correlation matrix
  • Heatmap visualization

5. Missing Values

  • Missing value matrix
  • Missing value heatmap
  • Nullity correlation

6. Sample (sample data)

  • First 10 rows
  • Last 10 rows
  • Random sample

7. Alerts

Automatically detected issues:

  • ⚠️ High correlation (> 0.9)
  • ⚠️ High missing rate (> 50%)
  • ⚠️ Constant/Quasi-constant features
  • ⚠️ High cardinality (categorical variables)
  • ⚠️ Imbalanced classes
  • ⚠️ Outliers

Example: Credit card fraud detection data

python generate_profile.py \
  --data-path "samples/datascience/data/raw/creditcard.csv" \
  --target-column "Class" \
  --mode explorative \
  --output-dir "outputs/reports"

Expected output:

═══════════════════════════════════════════════════════════
데이터 프로파일링 시작
═══════════════════════════════════════════════════════════

✓ 데이터 로드 완료: 284,807건, 31개 컬럼
✓ 메모리 사용량: 67.4 MB

─────────────────────────────────────────────────────────
기본 정보
─────────────────────────────────────────────────────────

전체 거래 건수: 284,807건
특성 개수: 31개
결측치: 0개

클래스 분포:
  정상 거래 (Class 0): 284,315건 (99.83%)
  사기 거래 (Class 1): 492건 (0.17%)
  불균형 비율: 1:578

─────────────────────────────────────────────────────────
프로파일링 리포트 생성 중...
─────────────────────────────────────────────────────────
⏳ 약 5분 소요될 수 있습니다...

✓ 완료!
📊 리포트 저장 위치: outputs/reports/creditcard_profile_report.html

🌐 브라우저에서 리포트가 자동으로 열렸습니다.

⚠️  주요 발견사항:
   - 클래스 불균형: 1:578 (사기: 0.17%)
   - Amount 변수에서 이상치 탐지
   - Time 변수는 초 단위 경과 시간

💡 권고사항:
   1. 클래스 불균형 처리 필요 (SMOTE, Undersampling)
   2. Amount 변수 스케일링 권장 (StandardScaler, RobustScaler)
   3. Time 변수에서 시간대 특성 추출 고려
   4. V1-V28은 PCA 변환된 특성 (직접 해석 어려움)

다음 단계:
   /engineer-features: 특성 엔지니어링
   /handle-imbalance: 클래스 불균형 처리

Automatic browser opening feature

The script automatically detects the operating system and opens the browser with the appropriate command:

import platform
import subprocess

def open_in_browser(filepath):
    """운영체제에 맞게 브라우저에서 HTML 파일 열기"""
    system = platform.system()

    if system == 'Darwin':  # macOS
        subprocess.run(['open', filepath])
    elif system == 'Linux':
        subprocess.run(['xdg-open', filepath])
    elif system == 'Windows':
        subprocess.run(['start', filepath], shell=True)

Disable option:

# 브라우저 자동 오픈 안 함
python generate_profile.py \
  --data-path "data.csv" \
  --no-browser

Performance optimization

Large data (over 1 million records)

  1. Use sampling:

    --sample-size 50000

  2. Minimal mode:

    --mode minimal

  3. Combine:

    python generate_profile.py \
  --data-path "big_data.csv" \
  --sample-size 100000 \
  --mode minimal

Reduce memory usage

  • Analyze after removing unnecessary columns
  • Use Default mode instead of Explorative mode
  • Limit sample size to 10,000–50,000

Output file management

File naming convention

{dataset_name}_profile_report.html

Example:

  • creditcard_profile_report.html
  • customer_churn_profile_report.html

Git management

HTML reports are large, so .gitignoreadd to:

outputs/reports/*.html

Reference documentation

License

MIT License