update-study

Update Study - Academic-Quality Experiment Report Generation

This is a skill that analyzes experiment logs and generates an experiment report at the level of an academic paper. It includes matplotlib visualization, Executive Summary, implementation structure, result interpretation, improvement proposals, and LaTeX PDF conversion.

Core Features

1. Incremental Detection - Analyze only new experiments (skip experiments already documented)
2. Data Extraction - Extract metric time series from W&B/log files, structure into a pandas DataFrame
3. Automated Plot Generation - matplotlib-based comparison charts, trend panels, per-condition analysis charts
4. Iterative Writing Loop - Academic paper-level quality improvement loop (pass at >=80 points)
5. Zero Hallucination - Cross-validation of log levels
6. LaTeX PDF Export - LaTeX conversion including Figures and PDF generation

Usage

/update-study <log_path(s)> <study_md_path>

• log_path(s): Path to experiment log files (multiple allowed, separated by spaces, including W&B export CSV)
• study_md_path: Path to the study markdown file to update

Arguments Parsing

$ARGUMENTSIn .md , if the last argument is a file, it is parsed as the study file, and the rest as log files.

예시:
  /update-study logs/exp1.log results/study.md
  → log_files: ["logs/exp1.log"]
  → study_file: "results/study.md"

  /update-study logs/exp1.log logs/exp2.log memgen_ablation_study.md
  → log_files: ["logs/exp1.log", "logs/exp2.log"]
  → study_file: "memgen_ablation_study.md"

---

Report Template

The final report follows the structure below. It maintains an academic paper-level format.

# {Title} Experiment Report

**Author**: {name}  **Date**: {date}  **Project**: {project}

## Executive Summary

| Task | Key Metric | Baseline | Proposed | Difference |
|------|-----------|----------|----------|------------|
| {task_a} | {metric} | {baseline_val} | {proposed_val} | {delta} |
| {task_b} | {metric} | {baseline_val} | {proposed_val} | {delta} |

{1-2 sentence overall conclusion}

## 1. Implementation

### 1.1 Overview

{실험의 목적과 접근 방식을 Definition-First 원칙으로 기술}

### 1.2 Implementation Structure

{핵심 코드 구조를 코드 스니펫으로 제시}

```python
class ProposedModule(nn.Module):
    """모듈 설명 - 무엇을, 왜 하는지"""
    def __init__(self, config):
        ...
    def forward(self, x):
        ...

2. Experimental Results

2.1 {Task A}: {one-line summary of key results}

{results table + Figure references + interpretation}

Figure 1 interpretation: {patterns observed in the figure, meaning, differences compared to baseline}

2.2 {Task B}: {one-line summary of key results}

{same structure}

2.3 {analysis title}: {cross-analysis or correlation analysis}

{comparative analysis across multiple experiments/conditions}

2.4 {limitation analysis title}

{limitations found in the experiment results, analysis of results that differ from expectations}

2.5 Experimental Setup

Parameter	Value
Model	{model_name}
Dataset	{dataset}
Epochs	{N}
Learning Rate	{lr}
Batch Size	{batch_size}
Hardware	{gpu_info}

3. Proposed Improvements

3.1 {Method A}

Problem: {specific problem found in the current results, numerical evidence}

Solution: {proposed solution and rationale}

# 구현 예시
class ImprovedModule(nn.Module):
    def __init__(self, config):
        ...

Expected Effect: {expected improvement effect and rationale}

3.2 {Method B}

{same structure: Problem → Solution → Code → Expected Effect}

4. Limitations

• {limitation 1}: {specific description + scope of impact}
• {limitation 2}: {specific description + scope of impact}
• {limitation 3}: {specific description + scope of impact}

5. Conclusion

{summary of key findings, contributions, significance}

6. Next Experiments

E{N}: {experiment name}

• Tests: H{N}
• Config changes:

  {parameter}: {new_value}  # was: {old_value}
  

• Expected: {expected results if the hypothesis is correct}

E{N+1}: {experiment name}

• Tests: H{N+1}
• Config changes:

  {config_changes}
  

• Expected: {prediction}

References

• {list of related papers/materials}

---

## Workflow Overview

Phase 0: Incremental Detection ├── Scan logs/ ├── Parse existing study (identify already analyzed experiments) └── Generate list of new experiments → if none, output "No new experiments" and exit

Phase 1: File Verification ├── Check that log files exist └── Read study.md

Phase 2: Data Extraction (NEW) ├── Extract metric time series from log/W&B files ├── Structure into a pandas DataFrame └── Metrics: accuracy, loss, ratio, entropy, etc.

Phase 3: Plot Generation (NEW) ├── Call scripts/generate_plots.py ├── Training curve comparison (Filter OFF vs ON style) ├── Metric trend (multi-subplot panel) ├── Per-condition comparison charts └── Output: save PNGs to the figures/ directory

Phase 4: Interpretation + Writing (Extended) ├── Call experiment-interpreter agent ├── Generate Executive Summary comparison table ├── Write interpretation for each Figure ├── Proposed Improvements section (Problem → Solution → Code → Expected Effect) ├── Write Limitations, Conclusion, Next Experiments └── Writing Quality Loop (>=80 points, repeat up to 3 times)

Phase 5: Verification ├── Call experiment-verifier agent ├── Verify numerical accuracy └── Verify logical consistency

Phase 6: Export (Extended) ├── Finalize Markdown ├── Convert to LaTeX + include Figures (scripts/export_latex_pdf.py) ├── Generate PDF └── Fallback: scripts/export_pdf.py (existing pandoc/weasyprint method)

---

## Phase 0: Incremental Detection

### Step 0.1: 로그 디렉토리 스캔

1. Generate a list of all log files in the logs/ directory

   • Patterns: *_train.log, *_eval.log, *.log, *.csv (W&B export)
   • Sort by file modification time

2. Cross-check with the input log file list

### Step 0.2: 기존 Study 분석

Identify experiments already documented in the existing study.md:

1. Search for experiment header patterns:

   • ### E{N}: or ### Experiment:
   • ## Experiment {N}:

2. Extract log file references:

   • [*_train.log:*] source notation in the format
   • Source: filenames in the column

3. Build the list of documented experiments: documented_experiments = { "evolve_h_only_train.log", "cross_attn_train.log", ... }

### Step 0.3: 새 실험 결정

```python
new_experiments = set(input_logs) - set(documented_experiments)

if len(new_experiments) == 0:
    print("No new experiments to analyze")
    print(f"  Already documented: {len(documented_experiments)} experiments")
    exit()  # 종료
else:
    print(f"Found {len(new_experiments)} new experiment(s) to analyze:")
    for exp in new_experiments:
        print(f"  - {exp}")

---

Phase 1: File Verification

Step 1.1: File Check

1. 새 실험 로그 파일 존재 확인
   - 각 log_path에 대해 파일 존재 여부 확인
   - 존재하지 않으면 에러 메시지 출력 후 중단

2. study.md 파일 확인
   - 파일이 존재하면 Read tool로 전체 내용 읽기
   - 파일이 없으면 새로 생성할 것임을 안내

3. 로그 파일 요약 정보 추출 (빠른 스캔)
   - 실험 config 파일 경로
   - 실험 모드 (train/evaluate)
   - 최종 metric 라인 위치

Step 1.2: Analyze Existing study.md

기존 study.md에서 확인할 사항:
- 이미 기록된 실험 목록
- 비교 가능한 baseline 결과
- 미해결 가설 목록
- 계획된 실험 목록 (이번 실험이 기존 계획에 해당하는지)

---

Phase 2: Data Extraction

Extract and structure metric time-series data from log files and W&B exports.

Step 2.1: Extract Metric Time Series

1. 로그 파일 파싱
   - 정규표현식으로 epoch/step별 메트릭 추출
   - 패턴 예시:
     - "Epoch {N}: accuracy={X.XX}, loss={X.XXXX}"
     - "Step {N} | train_loss: {X.XXXX} | eval_acc: {X.XX}"
     - W&B CSV: 컬럼 기반 직접 로드

2. W&B API/CSV 지원
   - wandb export CSV 파일 자동 인식
   - 컬럼명 표준화 (step, epoch, metric_name)

Step 2.2: Structure into pandas DataFrame

# 추출 결과 구조
import pandas as pd

# 각 실험별 DataFrame
df = pd.DataFrame({
    'step': [...],
    'epoch': [...],
    'train_loss': [...],
    'eval_accuracy': [...],
    'learning_rate': [...],
    'grad_norm': [...],
    # 실험별 추가 메트릭
    'ratio': [...],        # 필터링 비율 등
    'entropy': [...],      # 분포 엔트로피 등
})

# 메타데이터
metadata = {
    'experiment_name': '...',
    'config_path': '...',
    'log_path': '...',
    'total_epochs': N,
    'final_metrics': {
        'accuracy': X.XX,
        'loss': X.XXXX,
    }
}

Step 2.3: Target Metrics

필수 메트릭 (로그에 존재하는 경우):
- accuracy (eval/test)
- loss (train/eval)
- learning_rate
- grad_norm

선택 메트릭 (실험 유형에 따라):
- ratio (필터링 비율, 압축 비율 등)
- entropy (분포 엔트로피)
- perplexity
- BLEU / ROUGE
- custom metrics (로그에서 자동 감지)

---

Phase 3: Plot Generation

scripts/generate_plots.pyto generate matplotlib-based visualizations.

Step 3.1: Chart Types

1. Training Curve 비교 (필수)
   - X축: epoch 또는 step
   - Y축: loss / accuracy
   - 여러 실험을 한 차트에 오버레이
   - 스타일: "Filter OFF vs Filter ON" 비교 형태
   - 범례, 그리드, 타이틀 포함

2. Metric Trend Panel (필수)
   - Multi-subplot 레이아웃 (2x2 또는 3x2)
   - 각 subplot: 하나의 메트릭 시계열
   - 예: loss, accuracy, grad_norm, learning_rate
   - 실험 간 동일 스케일 사용

3. 조건별 비교 차트 (조건부)
   - Bar chart: 최종 메트릭 비교 (실험별)
   - Grouped bar: 여러 메트릭을 실험별로 그룹화
   - 예: baseline vs proposed 최종 accuracy 비교

4. 분포 차트 (조건부)
   - Histogram: metric 분포 비교
   - Box plot: 실험 간 변동성 비교

Step 3.2: Script Invocation

python scripts/generate_plots.py \
  --data extracted_metrics.json \
  --output-dir figures/ \
  --experiments "exp1,exp2,baseline" \
  --plots training_curve,metric_panel,comparison_bar

Step 3.3: Output

figures/
  ├── training_curve_comparison.png    # Loss/accuracy 비교 곡선
  ├── metric_trend_panel.png           # Multi-subplot 메트릭 패널
  ├── final_metric_comparison.png      # Bar chart 최종 비교
  └── {custom_name}.png                # 실험별 추가 차트

Step 3.4: Figure Style Criteria

# 학술 논문 수준 matplotlib 설정
import matplotlib.pyplot as plt

plt.rcParams.update({
    'figure.figsize': (10, 6),
    'figure.dpi': 150,
    'font.size': 12,
    'font.family': 'serif',
    'axes.labelsize': 13,
    'axes.titlesize': 14,
    'legend.fontsize': 11,
    'xtick.labelsize': 11,
    'ytick.labelsize': 11,
    'axes.grid': True,
    'grid.alpha': 0.3,
    'lines.linewidth': 2.0,
    'figure.constrained_layout.use': True,
})

# 색상 팔레트 (colorblind-friendly)
COLORS = ['#0072B2', '#D55E00', '#009E73', '#CC79A7', '#F0E442', '#56B4E9']

---

Phase 4: Interpretation + Writing

Step 4.1: Call Experiment-Interpreter Agent

Task tool 사용:
- subagent_type: "experiment-interpreter"
- prompt:

  "다음 실험 로그와 생성된 Figure를 분석하여
   학술 논문 수준의 실험 보고서를 작성해주세요.

  ## 데이터
  {추출된 DataFrame 요약}

  ## 생성된 Figure 목록
  {figures/ 디렉토리의 PNG 파일 목록과 각 차트 설명}

  ## 로그 파일
  {각 로그 파일의 전체 경로}

  ## 기존 study.md 내용 (비교용)
  {기존 study.md의 결과 요약 테이블}

  ## 이전 검증 피드백 (있는 경우)
  {verifier의 feedback_summary - 첫 iteration에는 없음}

  ## 요구사항
  1. Report Template 구조를 따를 것
  2. Executive Summary 비교 테이블 생성
  3. 각 Figure에 대한 구체적 해석 작성
     - 패턴, 추세, 이상점 기술
     - baseline 대비 차이 설명
     - Figure 번호로 참조 (Figure 1, Figure 2, ...)
  4. Implementation Structure 섹션에 핵심 코드 스니펫 포함
  5. Proposed Improvements: Problem → Solution → Code → Expected Effect
  6. Limitations 섹션: 구체적 한계와 영향 범위
  7. Conclusion: 주요 발견 요약
  8. Next Experiments: 가설과 연결된 구체적 계획
  9. 모든 수치에 출처 표기 (source: filepath:L행번호)
  10. 각 가설은 falsifiable + prediction + falsification 포함
  11. 출력은 Report Template 형식의 markdown

  ## 출력 형식
  references/interpretation-template.md 템플릿을 따르되,
  Report Template의 전체 구조에 맞게 확장해주세요."

Step 4.2: Generate Executive Summary

## Executive Summary

| Task | Key Metric | Baseline | Proposed | Difference |
|------|-----------|----------|----------|------------|
| {각 실험 task별 핵심 비교 결과} |

생성 규칙:
- Baseline은 기존 study에 기록된 vanilla/default 결과
- Proposed는 이번 실험의 결과
- Difference는 절대값 + 방향 (+/-) 포함
- 가장 중요한 메트릭 우선 배치

Step 4.3: Write Figure Interpretations

각 Figure에 대해:

1. Figure 참조 삽입:
   ![Figure N: {descriptive caption}](figures/{filename}.png)

2. 해석 문단 작성 (Topic-First 원칙):
   **Figure N 해석**: {핵심 관찰}. {상세 설명}. {baseline 대비 차이}.
   {가능한 원인/메커니즘 분석}.

3. 해석 품질 기준:
   - 정량적: 수치로 차이를 기술
   - 비교적: baseline 또는 이전 실험과 대비
   - 인과적: 왜 이런 패턴이 나타나는지 가설 제시

Step 4.4: Write Proposed Improvements

각 개선안에 대해 4-part 구조:

### 3.N {Method Name}

**Problem**: {실험 결과에서 발견된 구체적 문제}
  - 수치 근거 포함 (source: filepath:L행번호)
  - Figure 참조 포함 (Figure N 참조)

**Solution**: {제안하는 해결 방법}
  - 기존 연구/이론적 근거 제시
  - 구현 방향 설명

```python
# 구현 예시 코드
class ImprovedComponent:
    ...

Expected Effect: {expected improvement effect}

• Quantitative prediction (e.g., "expected accuracy improvement of +2-3%p")
• Explanation of rationale

### Step 4.5: Writing Quality Loop

작성된 초안에 대해 품질 평가 수행:

Evaluation criteria (see references/quality-criteria.md, weights updated):

1. Definition-First (25점)

   • Are all technical terms defined in the form "X is Y"?
   • Are new concepts defined before use?

2. Topic-First Paragraphs (20점)

   • Does every paragraph begin with a key result/claim?
   • Can the paragraph content be understood by reading only the first sentence?

3. Compare-Contrast (15점)

   • Are new results compared with previous experiments?
   • Are the causes/interpretations of differences presented?

4. Insight Depth (15점)

   • Is there analysis of "why" beyond surface-level description?
   • Are there hypotheses for results that differ from expectations?

5. Minimal Adjectives (5점)

   • Are there no unnecessary modifiers?
   • Are specific numbers used instead of subjective expressions?

6. Figure Quality (10점) [NEW]

   • Does every Figure have an interpretation paragraph?
   • Does the Figure caption accurately describe the content?
   • Are Figure references correctly linked in the main text?

7. Next Steps Clarity (10점) [NEW]

   • Is the next experiment clearly connected to the hypothesis?
   • Are the Config changes specific?
   • Are expected results and rationale presented?

Total score: /100 Passing threshold: >= 80 points

### Step 4.6: Revision (점수 < 80인 경우)

1. [Critical: Definition Missing]

   • Create a list of undefined terms
   • Add an "X is Y" definition for each term

2. [Critical: Topic-Last Paragraph]

   • Restructure paragraphs: in the order of key point → explanation → rationale

3. [Warning: No Comparison]

   • Add a comparison table with previous experiments
   • Add a difference analysis paragraph

4. [Warning: Shallow Insight]

   • Add an analysis of "Why did these results occur?"
   • Strengthen the hypothesis

5. [Minor: Excessive Adjectives]

   • "significantly improved" → "+12.5%p"
   • "much faster" → "2.3x speedup"

6. [Warning: Figure Without Interpretation]

   • Add an interpretation paragraph to Figures without interpretation
   • Fix Captions that are inaccurate

7. [Warning: Vague Next Steps]

   • Clarify the connection to the hypothesis
   • Make Config changes specific

### Step 4.7: Iteration Control

Maximum iterations: 3

Iteration 1: Draft → Quality evaluation → Revision (if needed) → Score >= 80: Proceed to Phase 5 → Score < 80: Collect feedback

Iteration 2: Revised draft → Re-evaluation → Score >= 80: Proceed to Phase 5 → Score < 80: Collect feedback

Iteration 3: Final revision → Re-evaluation → Score >= 80: Proceed to Phase 5 → Score < 80: Proceed with the current best version + report issues

---

## Phase 5: Verification (experiment-verifier)

### Task Tool 호출

Use Task tool:

• subagent_type: "experiment-verifier"

• prompt:

  "Please verify the following study.md updates.

  Verification targets (newly added sections)

  {Content written in Phase 4}

  Original log file paths

  {Full path of each log file}

  Existing study.md (for checking whether there are changes)

  {Existing study.md content}

  List of generated Figures

  {Contents of the figures/ directory}

  Verification requirements

  1. Cross-check all numerical values against the original logs (directly verify file:line)
  2. Numerical accuracy of the Executive Summary table
  3. Verify that Figure interpretations match the actual data
  4. Verify the logical validity of the interpretations
  5. Verify the falsifiability of all hypotheses
  6. Verify the feasibility of the next experiment
  7. Verify whether existing results have changed
  8. Syntactic correctness of code snippets

  Output

  Please return a verification report in JSON format."

### 결과 처리

```python
if verdict == "PASS":
    # Phase 6로 진행
elif iteration < 3:
    # feedback_summary를 Phase 4로 전달
    # interpreter에게 수정 요청
else:
    # 최대 반복 도달
    # 현재 최선 버전 저장
    # 미해결 이슈 사용자에게 보고

---

Phase 6: Export

Step 6.1: Finalize Markdown

1. study.md 최종 내용 저장
2. [NEW] 태그가 포함된 섹션 확인
3. Figure 참조 경로 검증 (figures/*.png 존재 확인)
4. 상대 경로 정규화

Step 6.2: LaTeX conversion + PDF generation (preferred method)

# scripts/export_latex_pdf.py 사용 (Figure 포함 LaTeX 변환)
python scripts/export_latex_pdf.py \
  study.md \
  study.pdf \
  --figures-dir figures/ \
  --template academic

기능:
- Markdown → LaTeX 변환
- Figure 자동 삽입 (\includegraphics)
- 학술 논문 레이아웃 (twocolumn 옵션 가능)
- TOC (Table of Contents) 포함
- [NEW] 태그 시각적 강조
- 테이블 포맷팅 (booktabs 스타일)
- 코드 블록 문법 강조 (listings/minted)
- 참고문헌 섹션

Step 6.3: Fallback PDF conversion

# LaTeX 변환 실패 시 기존 방식 사용
python scripts/export_pdf.py study.md study.pdf

Fallback 순서:
1. scripts/export_latex_pdf.py (LaTeX + Figure, 최상의 품질)
2. scripts/export_pdf.py → pandoc + LaTeX (Figure 수동 포함)
3. scripts/export_pdf.py → weasyprint (HTML 기반)
4. Markdown만 저장 (PDF 변환 실패 시 경고)

Step 6.4: Completion report

Update Complete!
  Markdown: study.md
  PDF: study.pdf
  Figures: figures/ ({N}개 PNG)
  New experiments: {N}개
  Writing quality: {score}/100
  Hypotheses: {N}개
  Next experiments: {N}개

---

Document Update Rules

Append Only principle

1. Append Only: Add new sections after existing content. Do not modify existing content.
2. [NEW] tag: Add [NEW] tag to newly added experiments (remove in the next update)
3. Separator: Before a new experiment, insert a --- separator
4. Date notation: Experiment execution date (based on log timestamp)
5. Consistent format: Complies with Report Template + references/interpretation-template.md

[NEW] Tag handling

---

### [NEW] Experiment: {experiment_name} ({YYYY-MM-DD})

...

On the next /update-study execution:

1. Previously added [NEW] remove all tags
2. Only for newly added sections [NEW] assign tags

---

Progress Reporting

Report status to the user during execution:

[Phase 0] Incremental Detection...
  Scanned logs/: {N}개 파일
  Already documented: {M}개 실험
  New experiments: {K}개 발견

[Phase 1] File Verification...
  로그 파일 확인: {K}개
  study.md 읽기 완료

[Phase 2] Data Extraction...
  메트릭 추출: {N}개 시계열
  DataFrame 구조화 완료
  메트릭 종류: accuracy, loss, grad_norm, ...

[Phase 3] Plot Generation...
  Training curve comparison: figures/training_curve_comparison.png
  Metric trend panel: figures/metric_trend_panel.png
  Final metric comparison: figures/final_metric_comparison.png
  총 {N}개 Figure 생성

[Phase 4] Interpretation + Writing...
  Executive Summary 생성 완료
  Figure 해석 작성: {N}개
  Proposed Improvements: {N}개
  → Writing Quality Loop:
    Iteration 1: Score 72/100
      - Critical: Definition missing (2)
      - Warning: Figure without interpretation (1)
    Iteration 2: Score 85/100
      All critical issues resolved

[Phase 5] Verification...
  → Numerical: {verified}/{total}
  → Logic: {sound}/{total}
  → Figure consistency: {matched}/{total}
  → Verdict: PASS

[Phase 6] Export...
  LaTeX 변환 완료
  PDF 생성 완료: study.pdf

Complete!
  - New experiments: {experiment_names}
  - Writing quality: {score}/100
  - Figures: {N}개
  - Hypotheses: {N}개
  - Next experiments: {N}개

---

Quality Criteria Summary

Criterion	Weight	Description
Definition-First	25점	100% of terms defined, in the form "X is Y"
Topic-First	20점	90% of paragraphs use the conclusion-first structure
Compare-Contrast	15점	Comparison table + difference analysis required
Insight Depth	15점	Include "why" analysis + mechanism hypothesis
Minimal Adjectives	5점	Data-driven expressions, minimize adjectives
Figure Quality	10점	Figure interpretation + caption + reference links
Next Steps Clarity	10점	Hypothesis connection + config specificity + prediction

Overall Pass: >= 80점

---

Additional Resources

• references/interpretation-template.md - Experiment interpretation template
• references/quality-criteria.md - Detailed criteria for writing quality evaluation
• scripts/generate_plots.py - matplotlib-based chart generation utility
• scripts/export_latex_pdf.py - LaTeX conversion + figure-included PDF generation
• scripts/export_pdf.py - PDF conversion utility (Fallback)

---

Cautions

1. Log files are ground truth: Numbers not in the logs cannot be used
2. Append Only: Never modify previous results
3. Source for every number: (source: filepath:L행번호) required
4. Hypotheses must be falsifiable: Delete unverifiable hypotheses
5. Repeat up to 3 times: Writing Quality Loop + Verification each up to 3 times
6. New experiments first: Automatically skip experiments already documented
7. [NEW] tag: Clearly mark newly added content
8. Figure required: Include at least 1 Figure in every experiment report
9. Code snippet verification: Check syntax correctness of Implementation/Improvements code
10. LaTeX compatibility: Check escaping of special characters in Markdown (_, %, #, &, etc.)