This Skill is for processingChinese General Social Survey (CGSS), China Household Income Project (CHIP), Chinese Social Survey (CSS)and other large-scale survey data with astandardized cleaning workflow. It ensures that every step from codebook to usable panel data is auditable and reproducible.
Applicable Scenarios
- After downloading raw data from databases such as CGSS/CHIP/CSS
- When merging survey data from multiple years
- When constructing panel data
- During the data preparation stage of a paper
Core Principles
The core principle of survey data cleaning: raw datais read-only and not modified, all transformation operations are recorded in the processed/ folder, and provenance.json。
Three Key Rules：
- Raw data must not be modified directly—all operations are performed in the
processed/folder - Every transformation step must be documented—including variable names, coding, and processing methods
- Missing values must be clearly marked—missing values must not be defaulted to 0 or other meaningful values
Workflow
原始数据 → 导入检查 → 变量命名规范化 → 缺失值处理 → 异常值处理 → 编码统一 → 合并面板 → 输出报告
Step 1 · Raw Data Import Check
Required：
import pandas as pd
import numpy as np
# 读取原始数据
df = pd.read_stata("data/raw/cgss2021.dta")
# 基础检查
print(f"样本量: {len(df)}")
print(f"变量数: {len(df.columns)}")
print(f"\n缺失值概况:")
print(df.isnull().sum()[df.isnull().sum() > 0])
Check Items：
| Check Items | Expected Results |
|---|---|
| Sample size | Consistent with the codebook description |
| Number of variables | Consistent with the codebook description |
| Variables with extreme missingness | Variables with a missing rate >50% must be flagged |
| Data type | Numeric/character types consistent with the codebook |
Step 2 · Variable Naming Standardization
Unified naming rules(recommended):
{数据库缩写}_{年份}_{变量原名}
例如：cgss_2021_income, chip_2018_urban
Variable coding records(must be written to provenance.json）：
{
"variable": "cgss_2021_urban",
"description": "城乡户籍类型",
"original_codes": {"1": "城市", "2": "农村", "9": "未知"},
"transformed_codes": {"1": "urban", "2": "rural"},
"missing_handling": "9 → NaN"
}
Step 3 · Missing Value Handling
Missing types that must be explicitly handled：
| Missing Type | Common codes in survey data | Handling Method |
|---|---|---|
| True missing | . / NA |
Keep as NaN |
| Not applicable | 97/98/99 |
Keep as NaN, do not fill with 0 |
| Refused to answer | 97 |
Keep as NaN |
| Don't know | 98 |
Keep as NaN |
| Data missing | 99 |
Keep as NaN |
⚠️ Common Mistakes：
- ❌ Filling
97/98/99with 0 (changes the variable mean) - ❌ Treating missing values as valid values
- ❌ Deleting all observations containing missing values (wastes data)
Correct Approach：
# 识别调查数据中的特殊编码
missing_codes = [97, 98, 99, 999, 9999]
for var in categorical_vars:
df[var] = df[var].replace(missing_codes, np.nan)
Step 4 · Outlier Detection
Common Methods：
# 描述统计 + 分布检查
print(df[numeric_vars].describe(percentiles=[.01, .05, .25, .50, .75, .95, .99]))
# 极端值检测（IQR方法）
def detect_outliers_iqr(series, factor=3):
Q1 = series.quantile(0.25)
Q3 = series.quantile(0.75)
IQR = Q3 - Q1
lower = Q1 - factor * IQR
upper = Q3 + factor * IQR
return (series < lower) | (series > upper)
outliers = detect_outliers_iqr(df['income'])
print(f"极端值数量: {outliers.sum()}")
Handling Method：
| Outlier Type | Handling Method |
|---|---|
| Data entry error (obviously unreasonable values) | Correct or flag as missing |
| Reasonable extreme values | Keep; report median instead of mean |
| Outliers (winsorize) | Two-sided winsorize (1% and 99% percentiles) |
Winsorize example：
from scipy.stats import mstats
for var in ['income', 'education_years']:
df[var] = mstats.winsorize(df[var], limits=[0.01, 0.01])
Step 5 · Code harmonization (multi-year/multi-database merging)
Longitudinal merging (multi-year CGSS)：
# 确保变量名一致
cgss_2017 = pd.read_stata("data/raw/cgss2017.dta")
cgss_2021 = pd.read_stata("data/raw/cgss2021.dta")
# 重命名统一
cgss_2017 = cgss_2017.rename(columns={
"a2017_income": "income",
"a2017_urban": "urban"
})
cgss_2021 = cgss_2021.rename(columns={
"a2021_income": "income",
"a2021_urban": "urban"
})
# 合并
df_combined = pd.concat([cgss_2017, cgss_2021], ignore_index=True)
df_combined['year'] = 2017 # 添加年份变量
Cross-sectional merging (CHIP + CGSS)：
# 统一变量定义和编码
# CHIP: income_2018; CGSS: income_2021
# 需标注：不同数据库收入定义可能不同
Step 6 · Build panel data
Long-wide format conversion：
# 宽 → 长
df_panel = df.melt(
id_vars=['id', 'year'],
value_vars=['income', 'education', 'employment'],
var_name='variable',
value_name='value'
)
Step 7 · Output report
Required output：
- Data cleaning report（
data/processed/cleaning_report.md） - Variable definition table（
data/processed/variable_definition.csv） - Updated provenance.json
Cleaning report template：
## 数据清洗报告
**原始数据**：CGSS 2021 (N=10,000)
**清洗后数据**：processed/cgss_2021_clean.dta (N=9,847)
**清洗时间**：[日期]
### 样本变动
| 处理类型 | 样本数 | 原因 |
|----------|--------|------|
| 原始样本 | 10,000 | |
| 删除重复记录 | -15 | ID重复 |
| 核心变量缺失 | -138 | income/education 缺失 |
| 清洗后 | 9,847 | |
### 变量处理记录
| 变量名 | 原始编码 | 处理方式 | 备注 |
|--------|----------|----------|------|
| income | 99999→NaN | 特殊编码替换 | 拒绝回答 |
| urban | 1/2 | 保持 | 城市=1,农村=2 |
How to invoke
In the Claude Code chat window, enter:
/codebook-pass
Or the full Prompt:
按调查数据清洗Skill处理本项目的CGSS/CHIP数据：原始数据导入检查→变量命名规范化→缺失值处理→异常值检测→编码统一→构建面板→输出清洗报告和provenance.json
Important notice
- Raw data must never be modified—all operations are performed in the
processed/folder - Missing value handling mustspecify the meaning of each code, and must not be uniformly treated as "missing" and then filled with 0
- Survey questionnaires from different years may have subtle differences; when merging, you mustverify variable by variable