data-cleaning-and-visualization

Data Cleaning and Visualization

Global process collaboration constraints (long-conversation drift prevention)

• This skill must not be used as an isolated entry point. When the user requests a complete paper, Word generation, process continuation, or when the stage is uncertain, first return to paper-workflow-orchestrator to determine the current S0-S8 stage.
• Before starting or continuing the formal tasks of this skill, you must run:

  python .claude/skills/paper-workflow-orchestrator/scripts/workflow_guard.py --skill data-cleaning-and-visualization
  

• If the output is [WORKFLOW FAIL] or the report shows status != "PASS", stop this skill, and according to paper_output/qa/workflow_guard_report.json , backfill the prerequisite stages based on the failed items; do not continue from memory.
• This skill only writes paper_output/ artifacts within its own contract scope; after completion, you must return to paper-workflow-orchestrator to determine the next step, and use context-memory-keeper to record completed artifacts, blockers, and the next step.
• In a long conversation, if the context becomes long, the stage is uncertain, or the user invokes the skill separately, first run:

  python .claude/skills/paper-workflow-orchestrator/scripts/workflow_guard.py --status
  

  then read paper_output/qa/workflow_guard_report.json、paper_output/preflight_report.json、paper_output/input_manifest.json、paper_output/results/run_manifest.json and the upstream JSON contract of this skill, and continue according to the recommended_skill and next_action in the report.
• Before continuing the process, you must treat paper_output/context/workflow_memory.json as a long-term breakpoint record; if its current_step、next_step、recommended_skill and workflow_guard.py --status are inconsistent, the guard report shall prevail.
• After completing the artifacts of this skill each time, first return to paper-workflow-orchestrator or run workflow_guard.py --status, then update workflow memory:

  python .claude/skills/context-memory-keeper/scripts/update_workflow_memory.py
  

  After updating, read paper_output/context/workflow_memory.json / .mdto confirm that the next step and recommended skill have been recorded.

Execution contract

• Upstream input: prioritize reading paper_output/input_manifest.json、paper_output/step1/problem_analysis.json and paper_output/plan/model_route.json; the formal process only handles attachments marked in the manifest as raw_data and usable_for_modeling=true .
• Must output:paper_output/data_cleaned/load_report.json、paper_output/plan/data_plan.json、paper_output/plan/visualization_plan.json、paper_output/figure_index.json; when there is processable data, also output paper_output/data_cleaned/ and paper_output/figures/。
• Downstream handoff:quality-assurance-auditor audit data/chart evidence; the S7 writing plan directly references figure_index.json, the table index, and the results contract.tasks.json is for legacy/quickstart only.
• Recommended next step: after completing the data and chart plan, proceed to quality-assurance-auditor to generate the task list; for a complete paper goal, return to paper-workflow-orchestrator to determine subsequent stages.
• Failure fallback: if there is no processable data file, still try to generate the plan file based on the problem intent and model route; do not directly treat template charts as final real results.

This skill is used to automatically process raw data in mathematical modeling, execute a standardized cleaning process, and generate basic exploratory data analysis (EDA) charts. It aims to reduce the tedious steps of manual data processing and quickly obtain the statistical characteristics and distribution of the data.

Important positioning: scripts are code-level prompts

The data table structures, field names, unit conventions, and chart requirements of mathematical modeling competition problems are usually different, so this skill's scripts/ should not be understood as a fixed program common to all competition problems. Their core value is to provide high-quality data processing and chart generation templates: including input/output directories, cleaning steps, chart dimensions, color schemes, annotations, save paths, and paper citation conventions.

In real competition problems, you should first analyze the data format and modeling requirements of the current attachments, then reference scripts/ Modify the writing in it a second time, or have the Agent read these scripts and regenerate new code adapted to the current competition problem.

Features

1. Automatic data source discovery: Automatically scan the problem_files/(competition problem attachments) or crawled_data/(crawler data) directories.
2. Read diagnostic report: First run robust_loader.py, generate paper_output/data_cleaned/load_report.json, and record the xlsx/csv/json structure and PDF diagnostic conclusions; the script will prioritize reading paper_output/input_manifest.json, and skip result_template, problem statement documents, and attachments that cannot be used for modeling. PDF table extraction is only for diagnosis and is not directly regarded as trustworthy raw data.
3. Data and chart plan: Generate data_plan.json、visualization_plan.json and figure_index.json, serving as the chart evidence handoff sheet for subsequent QA and main text generation.
4. Intelligent cleaning：
   • Automatically identify and convert numeric columns.
   • Handle missing values (fill numeric types with the mean/median, fill categorical types with the mode).
   • Remove all-empty rows/columns.
   • Simple outlier flagging/handling.
5. Automatic visualization：
   • Numeric variables: histograms, box plots.
   • Categorical variables: bar charts.
   • Multivariate relationships: correlation heatmaps, scatterplot matrices.
   • Paper-level chart templates: prediction comparison charts, residual distribution charts, scheme/model comparison charts, sensitivity analysis charts, weight charts, ranking charts, heatmaps, clustering scatter plots.
6. Standardized output: All cleaned data and charts are saved uniformly under the paper_output/ directory, making them easy to call during subsequent paper writing.

Script list

This skill includes the following core scripts, located under the .claude/skills/data-cleaning-and-visualization/scripts/ directory:

• scripts/robust_loader.py

  • When to use: Before any formal data cleaning, modeling, or plotting, first diagnose whether the attachments are readable, which sheets/fields are usable, and whether the PDF needs manual table conversion.
  • What it does: Prioritize reading paper_output/input_manifest.json, and generate a structure report only for xlsx/xls/csv/tsv/json marked as raw_data ; for PDFs, only generate text/table diagnostics, and do not treat automatically extracted PDF results as trustworthy data; output paper_output/data_cleaned/load_report.json, and record the current input_manifest_sha256. After the manifest changes, the loader must be rerun.

• scripts/run_pipeline.py

  • When to use: When the user provides competition problem data or completes crawling and needs cleaning and plotting to be done automatically. This is the most commonly used auxiliary script.
  • What it does: Sequentially generate the data/chart plan, call the cleaning and plotting scripts, and generate complete results under paper_output/ .

• scripts/build_data_visualization_plan.py

  • When to use: When problem_analysis.json or model_route.jsonalready exists and it is necessary to first clarify "which data supports which questions, and which charts go where".
  • What it does: Read the competition problem analysis, model route, and existing data files, and output paper_output/plan/data_plan.json、paper_output/plan/visualization_plan.json and paper_output/figure_index.json。

• scripts/clean_data.py

  • When to use: When only data cleaning is needed, plotting is not needed, or custom cleaning logic is needed.
  • What it does: Read the raw data and output cleaned CSV/Excel files to paper_output/data_cleaned/。

• scripts/visualize_data.py

  • When to use：When there is already cleaned data and charts need to be regenerated.
  • What to do：Read the data under paper_output/data_cleaned/ and generate basic EDA charts to paper_output/figures/。

• scripts/paper_figure_templates.py

  • When to use：When the Agent needs to generate paper-level chart code, read this file first as a code template.
  • What to do：Provide function templates for prediction comparison, residual distribution, model/scheme comparison, sensitivity analysis, indicator weights, composite score ranking, heatmaps, scatter plots, etc. When real data cannot be read, return a structured placeholder status and do not generate PNGs that could pass as real results.

• scripts/generate_paper_figures_from_plan.py

  • When to use：When there is already visualization_plan.json and cleaned CSV, and you want to first generate a draft of paper-level charts.
  • What to do：Call according to the chart plan paper_figure_templates.py, write the real generatable charts to paper_output/figures/fig_*.png, and in figure_index.json record ok、placeholder and status. Any placeholder cannot pass the formal evidence gate.

Output structure

After running, the following will be generated under the paper_output directory:

paper_output/
├── plan/
│   ├── data_plan.json       # 数据字段、清洗任务与子问题链接
│   └── visualization_plan.json # 建议图表、图题、用途与输出路径
├── figure_index.json        # 图表计划索引，供 QA 和正文生成核对
├── data_cleaned/       # 清洗后的数据文件
│   ├── load_report.json # 附件读取诊断报告
│   ├── dataset1_cleaned.csv
│   └── ...
├── figures/            # 生成的可视化图表
│   ├── fig_q1_1.png     # 按 visualization_plan 生成的论文级图表草稿
│   ├── fig_q1_2.png
│   ├── dataset1/
│   │   ├── dist_column_A.png
│   │   ├── heatmap.png
│   │   └── ...
│   └── ...

Directory conventions (aligned with the project globally)

• Input data is preferentially placed in problem_files/(competition problem attachments) and crawled_data/(supplementary/crawled data).
• This skill only writes to paper_output/, and will not modify the original data files.
• data_plan.json and visualization_plan.json are handoff sheets, not fixed code. The Agent should regenerate or modify the real modeling code based on them and the current attachment structure.
• paper_figure_templates.py generates paper chart code templates. If the current competition problem already has real model output, the real result tables should be connected to these templates first, rather than directly treating the template charts as final results.
• figure_index.json In placeholder=true、ok=false、exists=false, failure messages, or empty files all indicate that the chart evidence is incomplete; formal writing must not continue merely because an index entry exists.

Upstream and downstream connections

• Usually followed by:quality-assurance-auditor(generate task list) → paper-micro-unit-generator(generate and merge).
• To continue to the paper draft: return to paper-workflow-orchestrator。

Constraints (must be followed)

• Memory Interaction (required):
  • After completing cleaning, you must call context-memory-keeper, recording the "data quality overview (sample size/missingness)" and "key chart paths" to Short-term Workbench。
• This skill is only allowed to read:problem_files/ and crawled_data/; only allowed to write:paper_output/。
• Before the formal process reads attachments, it must first generate paper_output/input_manifest.json and paper_output/data_cleaned/load_report.json; the Agent must not skip the manifest and directly restate PDF table contents or treat result*.xlsx as raw data.
• Any chart that needs to appear in the paper must be referenced from paper_output/figures/ , avoiding scattering in the root directory or attachment directory.
• If the user's goal is to "produce a complete paper draft", after this skill ends it must enter:quality-assurance-auditor or directly return to paper-workflow-orchestrator, otherwise there will be a broken link of "charts but no main text/main text but no task list".