ab-testing-analyzer

AB Testing Analyzer

A fully functional intelligent AB testing analysis tool, developed based on the AB testing module of "Data Analysis Brother Ga's Ten Talks".

🎯 Skill Overview

This skill provides a complete AB testing solution from experimental design to result analysis, supporting multiple statistical testing methods, user segmentation analysis, and visual report generation.

✨ Core Features

• 🧪 Complete AB Testing Process
  • Experimental design and sample size calculation
  • Random group assignment validation
  • Conversion rate and retention rate analysis
  • Statistical significance testing

• 📊 Comprehensive Statistical Methods

  • t-test (independent samples, paired samples)
  • Chi-square test (goodness-of-fit, independence)
  • Confidence interval estimation
  • Effect size calculation
  • Multiple comparison correction

• 👥 Intelligent User Segmentation

  • Value segmentation (high/low value customers)
  • Demographic segmentation
  • Behavioral pattern segmentation
  • Custom segmentation strategies
  • Interaction effect analysis

• 📈 Rich Visualization Features

  • Conversion rate comparison chart
  • Retention rate curve chart
  • User segmentation heatmap
  • Interaction effect visualization
  • Statistical test result chart

• 🔧 Advanced Analysis Features

  • Multivariate AB testing
  • Bayesian AB testing
  • Time series analysis
  • Robustness checks
  • Causal inference support

🚀 Quick Start

1. Environment Requirements

# 依赖包
pip install pandas numpy scipy matplotlib seaborn statsmodels

2. Basic Usage

from scripts.ab_test_analyzer import ABTestAnalyzer
from scripts.statistical_tests import StatisticalTests
from scripts.visualizer import ABTestVisualizer

# 初始化分析器
analyzer = ABTestAnalyzer()
stats_tests = StatisticalTests()
visualizer = ABTestVisualizer()

# 加载AB测试数据
data = analyzer.load_data('ab_test_data.csv')

# 基础转化率分析
conversion_results = analyzer.analyze_conversion(
    data,
    group_col='页面版本',
    conversion_col='是否购买'
)

# 统计显著性检验
t_test_result = stats_tests.t_test(
    data,
    group_col='页面版本',
    metric_col='是否购买'
)

# 生成可视化报告
fig = visualizer.plot_conversion_comparison(conversion_results)

3. Run Examples

# 快速测试
python quick_test.py

# 基础AB测试示例
python examples/basic_ab_test_example.py

# 高级分群分析示例
python examples/advanced_segmentation_example.py

# 综合分析示例
python examples/comprehensive_analysis_example.py

📁 Project Structure

ab-testing-analyzer/
├── SKILL.md                     # 技能详细文档
├── README.md                    # 使用指南 (本文件)
├── quick_test.py               # 快速功能测试
├── test_skill.py               # 完整测试套件
│
├── scripts/                    # 核心功能模块
│   ├── __init__.py
│   ├── ab_test_analyzer.py         # AB测试核心分析
│   ├── statistical_tests.py        # 统计检验模块
│   ├── segment_analyzer.py         # 用户分群分析
│   └── visualizer.py              # 可视化生成器
│
└── examples/                   # 示例和数据
    ├── sample_data/            # 样本数据
    │   ├── sample_ab_test_data.csv
    │   └── sample_user_segments.csv
    ├── basic_ab_test_example.py     # 基础AB测试示例
    ├── advanced_segmentation_example.py # 高级分群分析示例
    └── comprehensive_analysis_example.py # 综合分析示例

💡 Main Features

1. Basic AB Testing Analysis

Conversion Rate Analysis

# 计算各组转化率
conversion_rates = analyzer.calculate_conversion_rates(
    data,
    group_col='实验组别',
    conversion_col='转化状态'
)

# 计算提升率和置信区间
lift_analysis = analyzer.calculate_lift(
    conversion_rates,
    control_group='对照组',
    test_group='测试组'
)

Retention Rate Analysis

# 计算留存率
retention_rates = analyzer.calculate_retention_rates(
    data,
    group_col='实验组别',
    retention_col='retention_7'
)

# 留存率曲线可视化
fig = visualizer.plot_retention_curves(retention_rates)

2. Statistical Significance Testing

t-test

# 独立样本t检验
t_result = stats_tests.t_test(
    data,
    group_col='页面版本',
    metric_col='购买金额',
    test_type='independent'
)

# 配对样本t检验
paired_t_result = stats_tests.t_test(
    before_after_data,
    group_col='用户ID',
    metric_col='行为指标',
    test_type='paired'
)

Chi-square test

# 拟合优度检验
chi2_goodness = stats_tests.chi_square_test(
    observed_data,
    expected_data,
    test_type='goodness_of_fit'
)

# 独立性检验
chi2_independence = stats_tests.chi_square_test(
    data,
    group_col='实验组别',
    outcome_col='转化状态',
    test_type='independence'
)

Effect size calculation

# Cohen's d计算
cohens_d = stats_tests.cohens_d(
    data,
    group_col='实验组别',
    metric_col='转化状态'
)

# Cramer's V计算
cramers_v = stats_tests.cramers_v(data, group_col, outcome_col)

3. User Segmentation Analysis

Value segmentation

from scripts.segment_analyzer import SegmentAnalyzer

segment_analyzer = SegmentAnalyzer()

# 基于价值的用户分群
value_segments = segment_analyzer.value_based_segmentation(
    data,
    value_col='累计消费金额',
    n_tiers=3
)

# 分群转化率分析
segment_conversion = segment_analyzer.segment_conversion_analysis(
    data,
    segment_col='价值组别',
    group_col='实验组别',
    conversion_col='转化状态'
)

Interaction effect analysis

# 页面版本与用户特征的交互效应
interaction_analysis = segment_analyzer.interaction_analysis(
    data,
    group_col='页面版本',
    segment_col='价值组别',
    outcome_col='转化状态'
)

# 交互效应可视化
fig = visualizer.plot_interaction_effects(interaction_analysis)

4. Advanced Statistical Analysis

Bayesian AB testing

# 贝叶斯AB测试分析
bayesian_result = analyzer.bayesian_ab_test(
    data,
    group_col='实验组别',
    conversion_col='转化状态',
    prior='jeffreys'
)

# 计算获胜概率
win_probability = analyzer.calculate_win_probability(bayesian_result)

Multivariate testing

# 多变量AB测试 (MVT)
mvt_result = analyzer.multivariate_test(
    data,
    group_cols=['页面版本', '按钮颜色', '标题文案'],
    conversion_col='转化状态'
)

5. Visualization Reports

Basic charts

# 转化率对比图
fig = visualizer.plot_conversion_comparison(
    conversion_data,
    title='AB测试转化率对比'
)

# 置信区间图
fig = visualizer.plot_confidence_intervals(
    statistical_results,
    metric='转化率'
)

# 用户分群热力图
fig = visualizer.plot_segment_heatmap(
    segment_data,
    title='用户分群转化率热力图'
)

Interactive dashboard

# 生成交互式仪表板
dashboard = visualizer.create_interactive_dashboard(
    analysis_results,
    output_file='ab_test_dashboard.html'
)

📊 Data Format

AB test data format (ab_test_data.csv)

用户ID,实验组别,转化状态,留存状态,累计消费金额,性别,年龄,价值组别,设备类型
U001,测试组,是,TRUE,299.99,男,25,高价值,移动端
U002,对照组,否,FALSE,59.99,女,32,低价值,PC端
U003,测试组,是,TRUE,599.99,男,28,高价值,移动端
U004,对照组,否,FALSE,199.99,女,35,中价值,PC端

User segmentation data format (user_segments.csv)

用户ID,RFM分群,行为分群,人口统计分群,综合分群
U001,高价值,活跃用户,年轻男性,高价值年轻用户
U002,低价值,流失用户,成熟女性,需要唤醒用户

🎯 Application Scenarios

Product Optimization

• Webpage revamp effect evaluation
• Feature launch impact analysis
• User interface optimization testing
• Performance improvement validation

Marketing Campaigns

• Ad creative testing
• Promotion strategy evaluation
• Email marketing optimization
• Social media campaign analysis

Operational Strategies

• Pricing strategy testing
• Recommendation algorithm optimization
• User registration process improvement
• Customer Service Strategy Evaluation

⚙️ Advanced Configuration

Statistical Parameter Settings

# 设置显著性水平
analyzer.set_significance_level(alpha=0.05)

# 设置统计功效
analyzer.set_statistical_power(power=0.8)

# 设置多重比较校正方法
analyzer.set_multiple_comparison_correction(method='bonferroni')

Custom Segmentation Strategy

# 定义自定义分群规则
custom_segments = {
    'high_value': {'累计消费金额': (500, float('inf'))},
    'medium_value': {'累计消费金额': (100, 500)},
    'low_value': {'累计消费金额': (0, 100)}
}

# 应用自定义分群
segmented_data = segment_analyzer.apply_custom_segments(
    data,
    segment_rules=custom_segments
)

Advanced Visualization Configuration

# 设置图表风格
visualizer.set_style(style='seaborn', palette='viridis')

# 自定义图表配置
chart_config = {
    'figsize': (12, 8),
    'dpi': 300,
    'format': 'png',
    'style': 'professional'
}

fig = visualizer.plot_with_config(data, config=chart_config)

🐛 Frequently Asked Questions

Q: How do I determine the appropriate sample size?

A: Use the sample size calculation function to calculate the minimum sample size, taking into account effect size, significance level, and statistical power.

Q: Does a p-value less than 0.05 mean the result is significant?

A: A p-value less than 0.05 indicates that, assuming the null hypothesis is true, the probability of observing the current result or a more extreme result is less than 5%. It needs to be interpreted in conjunction with effect size and practical significance.

Q: How do I handle the multiple comparisons problem?

A: Use methods such as Bonferroni correction and FDR correction to adjust p-values and avoid false positives.

Q: When should Bayesian AB testing be used?

A: Consider using Bayesian methods when prior information is needed, the sample size is small, or probabilistic conclusions are desired.

📈 Performance Optimization

• Use vectorized operations to accelerate computation
• Implement incremental statistical updates
• Use parallel computing to process large data
• Cache computation results to avoid repeated calculations
• Optimize memory usage patterns

📚 Technical Principles

Foundations of Statistical Testing

Based on hypothesis testing theory, statistical significance of experimental results is determined by calculating test statistics and p-values.

Central Limit Theorem

In the case of large samples, the distribution of sample means approaches a normal distribution, providing a theoretical foundation for many statistical methods.

Bayesian Inference

Combine prior information and observed data to perform parameter estimation and hypothesis testing through the posterior distribution.

Multiple Comparison Correction

When multiple hypothesis tests are performed simultaneously, control the overall error rate to avoid an increase in false positive results.

🤝 Contribution Guidelines

Issues and Pull Requests are welcome to improve this skill.

📄 License

MIT License

---

🎉 Get Started

Now that you have learned about all the features of the AB testing analysis skill, you can start using it:

# 快速验证功能
python quick_test.py

# 运行示例
python examples/basic_ab_test_example.py

Enjoy your AB testing analysis journey! 🚀