The context window is the model's working memory: finite, expensive, and subject to attention dilution. Managing it means deciding what deserves space, in what form, and for how long.
Overview
Bigger windows tempt you to dump everything in — full documents, entire histories, all tool outputs. But models attend unevenly: information in the middle gets lost, irrelevant content distracts, and every token costs money and latency. The skill is curation: budget the window like a budget, put the most decision-relevant content where attention is strongest (start and end), and move the rest into retrieval or summaries.
When to use
- Prompts are getting long and outputs are getting worse or more expensive.
- The model ignores instructions buried in large contexts.
- Multi-turn agents accumulate history until they slow down or degrade.
- You're deciding between a bigger model and a better context strategy (try the strategy first).
Core concepts
- Context budgeting: allocate the window explicitly — e.g., 20% system/task, 30% retrieved evidence, 30% working state, 20% headroom. Measure actual usage per category.
- Positional bias: models attend best to the start (instructions) and end (most recent). Put critical instructions first, the immediate task last, and bulk reference material in the middle — or better, retrieve it on demand.
- Compression ladder: raw text → extractive summary → abstractive summary → key facts → embeddings. Move content down the ladder as its immediacy fades.
- Retrieval over stuffing: keep a large corpus outside the window; pull in only the passages relevant to the current step. RAG is context management.
- Working state: a compact, structured scratchpad (goals, decisions, current step) maintained separately from the raw history. Cheaper and more reliable than full history.
- Eviction policies: when the window fills, what goes? Oldest-first is simple; relevance-based is better. Define the policy; don't let it be accidental.
Practical workflow
- Measure: log token usage per request broken down by content type. You can't manage what you don't measure.
- Set a budget per category and enforce it in code — truncate, summarize, or retrieve when a category overflows.
- Restructure prompts: instructions first, task and latest context last, reference material retrieved per-step.
- Replace raw history with a maintained working state plus a short recency window.
- For long documents, retrieve-then-read: fetch relevant chunks per question instead of loading the whole document.
- Re-measure after changes: confirm quality held (or improved) while tokens dropped.
Context budget example (128k window):
System + task instructions: 4k (front-load, never evict)
Working state (structured): 6k (maintained, rewritten)
Retrieved evidence: 20k (per-step, relevance-ranked)
Recent history: 10k (recency window)
Headroom: rest (for generation)
Common pitfalls
- Dump-and-pray: loading everything "because the window is big." Attention dilution degrades quality before the window fills.
- Lost in the middle: critical instructions buried mid-context. Test with the instruction moved to the start; the difference is often dramatic.
- No measurement: guessing about token usage. Instrument it — surprises are common.
- Summarizing away specifics: compression that drops numbers, names, and commitments. Summaries must preserve decision-relevant detail.
- Retrieval as an afterthought: bolting RAG onto a stuffed-context design. Design retrieval-first for large corpora.
- Ignoring latency: long contexts slow generation (quadratic attention costs in many architectures). Shorter contexts are faster as well as cheaper.