R-optimizer

R-optimizer — R language empirical analysis optimization Skill

This Skill targets economics empirical analysis scenarios, optimizing R code'sexecution efficiencyandmemory usage. Covers: vectorized operations, parallel computing, large-scale panel data processing, regression acceleration.Make regressions that originally took hours complete in minutes.

Applicable scenarios

• Processing large survey data such as CGSS/CHIP (sample size > 50,000)
• Panel data fixed effects estimation (large number of dummy variables)
• Monte Carlo simulation or Bootstrap (requiring large amounts of repeated computation)
• Staggered DID event study (multi-time point estimation)
• Rapid iteration during the paper draft stage

Core philosophy

The bottleneck of empirical analysis is often not the algorithm, but I/O and loops. Optimization order:

1. Data reading and writing(fastest optimization point)
2. Vectorization(eliminate R-level loops)
3. Parallelization(multi-core simultaneous computation)
4. Memory management(reduce data copying)

Workflow

诊断慢点 → 向量化优化 → 并行化加速 → 内存优化 → 验证结果一致性

Step 1 · Diagnose slow points

Use profvis to locate bottlenecks：

# 安装并使用 profvis
install.packages("profvis")

library(profvis)
profvis({
  # 放入需要诊断的代码
  df <- readstata13::read.dta13("data/raw/cgss2021.dta")
  result <- lm(income ~ education + age + urban, data = df)
})

Common bottleneck identification：

Code pattern	Problem	Optimization direction
for (i in 1:nrow(df))	Row loops	Vectorization
merge() multiple times	I/O bottleneck	data.table::merge
lm(y ~ x1 + x2 + ... + x100)	Large matrix inversion	Fixed effects projection matrix
boot() large amounts of repetition	Single-threaded	Parallelization

Step 2 · Vectorization optimization

Use data.table instead of data.frame：

library(data.table)

# 读取速度：data.table 比 read.csv 快10-100倍
DT <- fread("data/raw/cgss2021.csv")

# 分组计算：避免 for 循环
DT[, .(mean_income = mean(income, na.rm = TRUE),
       sd_income = sd(income, na.rm = TRUE)),
   by = .(urban, year)]

# 滚动窗口：向量化替代循环
DT[, lag_income := shift(income, 1), by = id]
DT[, income_growth := income / lag_income - 1]

Example of replacing loops with vectorization：

# ❌ 慢：行循环
for (i in 2:nrow(df)) {
  df$income_growth[i] <- (df$income[i] - df$income[i-1]) / df$income[i-1]
}

# ✅ 快：向量化
library(data.table)
DT <- as.data.table(df)
DT[, income_growth := income / shift(income, 1) - 1, by = id]

Step 3 · Parallelization acceleration

Use future + furrr to implement parallelism：

# 安装
install.packages(c("future", "furrr", "doParallel"))

library(future)
library(furrr)
plan(multisession, workers = 6)  # 使用6个核心

# 并行化回归：对多个因变量分别回归
library(furrr)
library(broom)

models <- c("income", "education", "employment")

results <- future_map(models, ~ {
  lm(as.formula(paste(.x, "~ treatment + controls")), data = df) %>%
    tidy() %>%
    filter(term == "treatment")
})

Parallel Bootstrap：

library(doParallel)
cl <- makeCluster(6)
registerDoParallel(cl)

boot_results <- foreach(i = 1:1000, .combine = rbind) %dopar% {
  # Bootstrap 抽样
  idx <- sample(nrow(df), replace = TRUE)
  df_boot <- df[idx, ]
  
  # 单次回归
  coef(lm(income ~ treatment, data = df_boot))
}

stopCluster(cl)

Step 4 · Large-scale panel data processing

Use fixest instead of felm or manual fixed effects：

# 安装
install.packages("fixest")

library(fixest)

# 固定效应回归：比 lm 快了 10-100倍
# 原理：先做组内变换（within transformation），不构造大矩阵

# 双向固定效应（国家+年份）
feols(income ~ treatment | country + year,
      data = DT,
      cluster = ~country)  # 聚类标准误

# 聚类在多个层面（双向聚类）
feols(income ~ treatment | country^year,
      data = DT,
      cluster = ~country + year)

# 多时点 DID（交错处理）
feols(log_gdp ~ treatment | country + year,
      data = DT,
      panel.id = ~country + year,
      didsetup = TRUE)  # 自动识别交错处理

# 输出带星标的回归表
etable(feols_model1, feols_model2,
       digits = 3,
       signifCode = FALSE,
       tex = TRUE)

fixest vs lm performance comparison：

Scenario	lm	felm (lfe)	fixest
100 countries × 30 years panel	Slow	Fast	Fastest
Clustered standard errors (country level)	✅	✅	✅
Two-way fixed effects	✅	✅	✅
Parallel computing	❌	❌	✅

Step 5 · Memory optimization

Reduce data copying：

# ❌ 每次操作都拷贝数据
df2 <- df[df$year > 2010, ]
df3 <- df2[df2$urban == 1, ]

# ✅ 用 data.table 原地修改
DT <- fread("data/raw/cgss2021.csv")
setkey(DT, year)
DT <- DT[J(2011:2021)]  # 原地过滤，不拷贝

Large-scale data reading and writing：

# ❌ read.csv 慢且占内存
df <- read.csv("data/raw/large_file.csv")

# ✅ fread 自动检测类型，速度快10倍
DT <- fread("data/raw/large_file.csv")

# ✅ 对超大型文件（>10GB）分块读取
DT <- fread("data/raw/large_file.csv",
            nrows = 100000,  # 先读10万行
            skip = 1000000)  # 跳过前100万行

Step 6 · Verify result consistency

After optimization, result consistency must be verified：

# 验证：向量化版本 vs 循环版本结果一致
original_result <- loop_version(df)
vectorized_result <- vectorized_version(df)

all.equal(original_result, vectorized_result)  # 必须返回 TRUE

# 验证：并行版本 vs 单线程版本结果一致
set.seed(42)
parallel_boot <- parallel_bootstrap(df, nboot = 1000, ncores = 6)

set.seed(42)
sequential_boot <- sequential_bootstrap(df, nboot = 1000)

max_abs_diff <- max(abs(parallel_boot - sequential_boot))
print(paste("最大差异:", max_abs_diff))  # 必须 < 1e-6

Output specification

# 优化代码保存规范
# 保存脚本：scripts/optimize_regression.R
# 主脚本：analysis/01_baseline_regression.R
# 输出：output/tables/regression_table1.tex

Invocation interface

Enter in the Claude Code conversation window:

/R-optimizer

Or the full Prompt:

用R-optimizer Skill优化本项目的R代码：使用data.table向量化处理、fixest固定效应回归、并行Bootstrap。确保优化后结果与优化前一致。

Coordination with other Skills

• codebook-pass → Process cleaned data with R optimization scripts
• latex-table → fixest 's etable() directly outputs LaTeX
• did-reviewer → DID estimation uses fixest 's multi-time point functionality

Important Statement

• Verify that results are consistent after optimization—Efficiency gains should not come at the expense of correctness
• Do not optimize prematurely—First get the code running, then optimize for bottlenecks
• Parallelization has overhead—For small samples (n<5000), parallelization is not recommended; the gains are less than the overhead