ai-researcher

AI Researcher

When to Use

• Surveying state-of-the-art models, methods, or benchmarks
• Comparing model families or techniques with fair experimental design
• Designing ablation studies with controlled variables
• Writing research memos or technical reports for stakeholder decisions
• Critiquing methodology in papers or internal experiments
• Planning novel experiments with falsifiable hypotheses
• Reproducing published results and verifying claims

When NOT to Use

• Shipping production LLM features, RAG, or agent systems → ai-engineer
• Enterprise AI policy, regulation, or risk tiering → ai-risk-governance
• Adversarial product testing or jailbreak campaigns → ai-redteam
• Classical ML pipelines, A/B testing, or statistical analysis → data-scientist

Related skills

Need	Skill
Production RAG, agents, deployment	ai-engineer
Prompt and agent implementation detail	prompt-engineer
Classical ML and A/B statistics	data-scientist
Governance, regulation, risk registers	ai-risk-governance
Red-team attacks on deployed systems	ai-redteam
Token/context efficiency research	research-engineer-scientist-tokens
Safeguard ML benchmarks and classifiers	ml-research-engineer-safeguards
RL distributed training infrastructure	ml-systems-engineer-rl-engineering

Core Workflows

1. Research question framing

1. Convert vague ask into falsifiable question
2. Define scope: task, data regime, compute budget, timeline
3. List baselines that must be beaten or matched
4. Specify primary and secondary metrics
5. Document assumptions and out-of-scope items

See references/research_framing.md for question templates and hypothesis types.

2. Literature review

Process:

1. Search: arXiv, ACL Anthology, OpenReview, major labs' blogs
2. Screen by relevance, recency, citation quality
3. Extract: problem, method, data, metrics, limitations
4. Synthesize themes and open gaps
5. Cite primary sources; avoid over-relying on secondary summaries

See references/literature_review.md for screening matrix and synthesis outline.

3. Experimental design

Element	Requirement
Baselines	Strong and fair (same data, tuning budget)
Ablations	One change at a time
Seeds	Multiple runs for stochastic methods
Stats	Confidence intervals, not single-point luck
Reproducibility	Config, data version, code commit logged

See references/experiment_design.md for power analysis pointers and leakage checks.

4. Benchmarking and analysis

• Use public benchmarks when task-aligned; document train/test contamination risk
• Report compute cost (GPU hours) alongside accuracy
• Separate in-distribution vs stress tests
• Visualize failure modes, not only aggregate scores

See references/benchmarking.md for leaderboard caveats and custom eval sets.

5. Research communication

Deliverable types: memo (1–3 pages), technical report, slide deck for decision meeting.

Include: question, method summary, results table, limitations, recommended next step.

See references/research_writing.md for memo structure and peer-review checklist.

When to load references

• Question and hypothesis → references/research_framing.md
• Literature survey → references/literature_review.md
• Experiments → references/experiment_design.md
• Benchmarks → references/benchmarking.md
• Writing → references/research_writing.md