data-cleaning-and-visualization

Data Cleaning and Visualization

Global workflow collaboration constraints (long-conversation drift prevention)

• This skill must not be used as an isolated entry point. When the user requests a complete paper, Word generation, continued workflow, or is unsure of the stage, first return to paper-workflow-orchestrator to determine the current S0-S8 stage.
• Before starting or continuing a formal task of this skill, you must run:

  python .claude/skills/paper-workflow-orchestrator/scripts/workflow_guard.py --skill data-cleaning-and-visualization
  

• If the output is [WORKFLOW FAIL] or reports status != "PASS", stop this skill, and backfill the prerequisite stages according to the failure items in paper_output/qa/workflow_guard_report.json ; do not continue from memory.
• This skill only writes the paper_output/ artifacts within its own contract scope; after completion, you must return to paper-workflow-orchestrator to determine the next step, and use context-memory-keeper to record completed artifacts, blockers, and next steps.
• In long conversations, if the context grows long, the stage is uncertain, or the user invokes skills separately, first run:

  python .claude/skills/paper-workflow-orchestrator/scripts/workflow_guard.py --status
  

  then read paper_output/qa/workflow_guard_report.json、paper_output/preflight_report.json、paper_output/input_manifest.json、paper_output/results/run_manifest.json and this skill's upstream JSON contract, and continue according to the recommended_skill and next_action in the report.
• Before continuing the workflow, you must treat paper_output/context/workflow_memory.json as a long-term breakpoint record; if its current_step、next_step、recommended_skill and workflow_guard.py --status are inconsistent, the guard report shall prevail.
• After completing this skill's artifacts each time, first return to paper-workflow-orchestrator or run workflow_guard.py --status, then update workflow memory:

  python .claude/skills/context-memory-keeper/scripts/update_workflow_memory.py
  

  after updating, read paper_output/context/workflow_memory.json / .md, and confirm that the next step and recommended skill have been recorded.

Execution contract

• Upstream input: preferentially read paper_output/input_manifest.json、paper_output/step1/problem_analysis.json and paper_output/plan/model_route.json; the formal workflow only processes attachments marked in the manifest as raw_data and usable_for_modeling=true .
• Must output:paper_output/data_cleaned/load_report.json、paper_output/plan/data_plan.json、paper_output/plan/visualization_plan.json、paper_output/figure_index.json; when there is processable data, also output paper_output/data_cleaned/ and paper_output/figures/。
• Downstream handoff:quality-assurance-auditor reads the data/chart contract to complete tasks.json；paper-micro-unit-generator references chart evidence through the task list.
• Recommended next step: after completing the data and chart plan, proceed to quality-assurance-auditor to generate the task list; for a complete paper goal, should return to paper-workflow-orchestrator to determine subsequent stages.
• Failure fallback: if there is no processable data file, still try to generate the plan file based on the problem statement and model route; do not directly treat template charts as the final real results.

This skill is used to automatically process raw data in mathematical modeling, execute a standardized cleaning workflow, and generate basic exploratory data analysis (EDA) charts. It aims to reduce the tedious steps of manual data processing and quickly obtain the statistical characteristics and distribution of the data.

Important positioning: scripts are code-level prompts

The data table structures, field names, unit conventions, and chart requirements of mathematical modeling competition problems are usually different, so this skill's scripts/ should not be understood as a fixed procedure common to all competition problems. Their core value is to provide high-quality templates for data processing and chart generation: including input/output directories, cleaning steps, chart dimensions, color schemes, annotations, save paths, and paper citation conventions.

In real competition problems, you should first analyze the data format and modeling requirements of the current attachment, then reference scripts/ Or let the Agent read these scripts and regenerate new code adapted to the current competition problem.

Features

1. Automatic data source discovery: Automatically scan the problem_files/(competition problem attachments) or crawled_data/(crawler data) directory.
2. Read diagnostic report: First run robust_loader.py, generate paper_output/data_cleaned/load_report.json, recording the xlsx/csv/json structure and PDF diagnostic conclusions; the script will preferentially read paper_output/input_manifest.json, and skip result_template, problem statement documents, and attachments that cannot be used for modeling. PDF table extraction is only for diagnosis and is not directly regarded as trustworthy raw data.
3. Data and chart plan: Generate data_plan.json、visualization_plan.json and figure_index.jsonas the chart evidence handoff sheet for subsequent QA and main text generation.
4. Intelligent cleaning：
   • Automatically identify and convert numeric columns.
   • Handle missing values (fill numeric types with mean/median, and categorical types with mode).
   • Remove fully empty rows/columns.
   • Simple outlier marking/handling.
5. Automatic visualization：
   • Numeric variables: histograms, box plots.
   • Categorical variables: bar charts.
   • Multivariate relationships: correlation heatmaps, scatterplot matrices.
   • Paper-level chart templates: prediction comparison charts, residual distribution charts, scheme/model comparison charts, sensitivity analysis charts, weight charts, ranking charts, heatmaps, clustering scatter plots.
6. Standardized output: All cleaned data and charts are saved uniformly under the paper_output/ directory, making it convenient for subsequent paper writing calls.

Script List

This skill includes the following core scripts, located under the .claude/skills/data-cleaning-and-visualization/scripts/ directory:

• scripts/robust_loader.py

  • When to use: Before any formal data cleaning, modeling, or plotting, first diagnose whether the attachments are readable, which sheets/fields are usable, and whether PDFs need manual table conversion.
  • What it does: Preferentially read paper_output/input_manifest.json, and generate structure reports only for xlsx/xls/csv/tsv/json marked as raw_data ; for PDFs, generate only text/table diagnostics, and do not treat automatically extracted PDF results as trustworthy data; output paper_output/data_cleaned/load_report.json。

• scripts/run_pipeline.py

  • When to use: When the user provides competition problem data or has completed crawling and needs automatic cleaning and plotting. This is the most commonly used auxiliary script.
  • What it does: Sequentially generates the data/chart plan, calls the cleaning and plotting scripts, and generates complete results under paper_output/ .

• scripts/build_data_visualization_plan.py

  • When to use: When problem_analysis.json or model_route.jsonalready exists, and it is necessary to first clarify “which data supports which questions, and which charts go where.”
  • What it does: Read the competition problem analysis, model route, and existing data files, and output paper_output/plan/data_plan.json、paper_output/plan/visualization_plan.json and paper_output/figure_index.json。

• scripts/clean_data.py

  • When to use: When only data cleaning is needed, plotting is not needed, or custom cleaning logic is required.
  • What it does: Read raw data and output cleaned CSV/Excel files to paper_output/data_cleaned/。

• scripts/visualize_data.py

  • When to use: When cleaned data is already available and charts need to be regenerated.
  • What it does: Read paper_output/data_cleaned/ data under, generate basic EDA charts to paper_output/figures/。

• scripts/paper_figure_templates.py

  • When to use: When the Agent needs to generate paper-level chart code, prioritize reading this file as a code template.
  • What it does: Provides function templates for prediction comparison, residual distribution, model/scheme comparison, sensitivity analysis, indicator weights, composite score ranking, heatmaps, scatter plots, etc.

• scripts/generate_paper_figures_from_plan.py

  • When to use: When you already have visualization_plan.json and cleaned CSV, and want to first generate a draft of paper-level charts.
  • What it does: Calls paper_figure_templates.pyaccording to the chart plan, generates the planned charts to paper_output/figures/fig_*.png, and updates paper_output/figure_index.json。

Output structure

After running, the following content will be generated under the paper_output directory:

paper_output/
├── plan/
│   ├── data_plan.json       # 数据字段、清洗任务与子问题链接
│   └── visualization_plan.json # 建议图表、图题、用途与输出路径
├── figure_index.json        # 图表计划索引，供 QA 和正文生成核对
├── data_cleaned/       # 清洗后的数据文件
│   ├── load_report.json # 附件读取诊断报告
│   ├── dataset1_cleaned.csv
│   └── ...
├── figures/            # 生成的可视化图表
│   ├── fig_q1_1.png     # 按 visualization_plan 生成的论文级图表草稿
│   ├── fig_q1_2.png
│   ├── dataset1/
│   │   ├── dist_column_A.png
│   │   ├── heatmap.png
│   │   └── ...
│   └── ...

Directory conventions (aligned with the project globally)

• Input data should preferentially be placed in problem_files/(competition attachments) and crawled_data/(supplementary/crawled data).
• This skill only writes to paper_output/, and will not modify raw data files.
• data_plan.json and visualization_plan.json are handoff sheets, not fixed code. The Agent should regenerate or modify the actual modeling code based on them and the current attachment structure.
• paper_figure_templates.py generates paper chart code templates. If the current competition problem already has real model outputs, the real result tables should be connected to these templates first, rather than directly treating the template charts as final results.

Upstream and downstream connections

• Usually followed by:quality-assurance-auditor(generate task list) → paper-micro-unit-generator(generate and merge).
• To continue to the paper draft: return to paper-workflow-orchestrator。

Constraints (must be followed)

• Memory Interaction (required):
  • After completing cleaning, you must call context-memory-keeper, record the "data quality overview (sample size/missingness)" and "key chart paths" to Short-term Workbench。
• This skill is only allowed to read:problem_files/ and crawled_data/; only allowed to write:paper_output/。
• Before the formal process reads attachments, it must first generate paper_output/input_manifest.json and paper_output/data_cleaned/load_report.json; the Agent must not skip the manifest and directly restate PDF table content or treat result*.xlsx as raw data.
• Any chart that needs to appear in the paper must be referenced from paper_output/figures/ , avoiding scattering them in the root directory or attachment directory.
• If the user's goal is to "produce a complete paper draft", after this skill ends you must enter:quality-assurance-auditor or directly return to paper-workflow-orchestrator, otherwise there will be a broken link of "charts but no main text/main text but no task list".