When to Use
- Frame research questions on tokens, context length, or inference cost
- Design experiments with baselines, ablations, and statistical rigor
- Build benchmarks for tokens-per-successful-task, effective context, cache leverage
- Measure tokenizer and formatting effects on length and model behavior
- Evaluate compression, summarization, routing, or distillation for token savings
- Analyze long-context phenomena (needle, lost-in-middle, attention budget)
- Write research memos with reproducible methods and honest limitations
- Translate findings into actionable thresholds for engineering and product
When NOT to Use
- Executive token reduction program with phased rollout →
ai-token-improvement-plan-engineer - Implement context packing, compaction code paths →
ai-context-engineer - Rewrite one production prompt →
prompt-engineer - General literature survey unrelated to tokens →
ai-researcher - Production RAG/agent deployment →
ai-engineer - Classical ML without LLM token focus →
data-scientist
Related skills
| Need | Skill |
|---|---|
| General research methodology | ai-researcher |
| Cost improvement program / roadmap | ai-token-improvement-plan-engineer |
| Production context assembly | ai-context-engineer |
| Prompt wording and eval harness | prompt-engineer |
| RAG and agent runtime build | ai-engineer |
| Statistical testing and cohort analysis | data-scientist |
| Adversarial robustness of compressed context | ai-redteam |
| Commercial AI architecture | applied-ai-architect-commercial-enterprise |
Core Workflows
1. Research framing (tokens)
Hypothesis, metrics, baselines, budget.
See references/research_framing_tokens.md.
2. Measurement and instrumentation
Token accounting, logging, fair comparison.
See references/measurement_instrumentation.md.
3. Experiment design and ablations
Controls, sweeps, power, stopping rules.
See references/experiment_design_ablations.md.
4. Context, tokenization, and long-context
Tokenizer, placement, window effects.
See references/context_tokenization_longcontext.md.
5. Compression and efficiency methods
Summarization, routing, distillation research.
See references/compression_efficiency_methods.md.
6. Reproducibility and research reporting
Memos, artifacts, handoff to engineering.
See references/reproducibility_reporting.md.
Outputs
- Pre-registration / experiment plan — hypothesis, metrics, stop criteria
- Results table — mean ± CI; tokens and quality side by side
- Pareto chart narrative — quality vs tokens at operating points
- Ablation appendix — what mattered, what did not
- Research memo — conclusion, limits, recommended next build
- Artifact bundle — configs, seeds, eval scripts, hashed datasets
Principles
- Report tokens and quality together — never optimize one without the other
- Match tokenizer and model — counts from the deployment tokenizer/API
- Control confounds — temperature, system prompt, tool schemas held fixed across arms
- Pre-register primary metric — avoid p-hacking across slice metrics
- Separate science from rollout — research recommends;
ai-token-improvement-plan-engineerowns program