reproducibility-audit

Reproducibility Audit

Purpose

Help the user make a research project reproducible enough that another person, or their future self, can rerun it. This skill follows the handbook's reproducibility section: exact environments, data and code versioning, documentation, random seeds, parameter logging, and backups.

The output is an audit report plus a prioritized fix list.

When to Use

• User is starting or cleaning a research codebase
• User needs collaborator handoff
• User cannot reproduce an old experiment
• User is preparing for paper submission or public release
• User is moving between local, server, HPC, Docker, or conda environments

Workflow

Stage 1: Identify the Project Context

Ask:

• What repository or folder is being audited?
• Is this local, remote server, HPC, or containerized?
• Is the goal internal reproducibility, collaborator onboarding, or public release?
• What result must be reproducible?

If the user grants filesystem access, inspect the repo structure before advising.

Stage 2: Audit the Three Pillars

Check:

Environment management

• Conda/uv/pip environment file exists
• Python/CUDA/framework versions are pinned where needed
• Setup instructions are current
• Dockerfile or container instructions exist if portability matters

Data and code versioning

• Git tracks source and config, not generated clutter
• Dataset source/version/preprocessing are documented
• Large files have an intentional storage path
• Checkpoints and outputs are named consistently

Documentation

• README explains install, data setup, train/eval commands
• Commands are copy-pasteable
• Expected outputs and runtime are stated
• Known caveats and hardware assumptions are explicit

Stage 3: Audit Experiment Hygiene

Check:

• Random seeds
• Config files
• Hyperparameter logging
• Commit hash logging
• Metric files
• Plot/table generation scripts
• Failure logs
• Result-to-paper traceability

Stage 4: Classify Issues

Use severity:

• Blocker: current results cannot be rerun or verified
• High: collaborators will likely fail to run it
• Medium: results can run but are hard to inspect or compare
• Low: polish or future-proofing

Prioritize fixes that protect irreplaceable results first.

Stage 5: Produce the Artifact

Save to ~/phd-log/reproducibility/YYYY-MM-DD-[project].md.

# Reproducibility Audit — [Project]

## Goal
[Internal / collaborator / release] reproducibility for [target result]

## Summary
- Overall status:
- Biggest risk:
- First fix:

## Environment
| Check | Status | Notes |
|---|---|---|

## Data and code versioning
| Check | Status | Notes |
|---|---|---|

## Documentation
| Check | Status | Notes |
|---|---|---|

## Experiment hygiene
| Check | Status | Notes |
|---|---|---|

## Issues
| Severity | Issue | Fix |
|---|---|---|

## Prioritized action plan
- [ ] Blocker:
- [ ] High:
- [ ] Medium:
- [ ] Low:

Tone

Be practical. A reproducibility audit should leave the user with 3-5 high-impact fixes, not a guilt-inducing wall of best practices.

What Not to Do

• Do not demand Docker if conda or uv is enough for the collaboration context.
• Do not recommend version pinning without explaining what needs to be pinned.
• Do not ignore data provenance.
• Do not treat README polish as more urgent than rerunnable experiments.