The most expensive tokens in agent work are the ones nobody reads: 300 identical JSON rows when the schema plus three samples would do, a log where one error hides among four hundred heartbeats, a file pasted whole for one relevant section. This skill crushes those structurally — schema + head/tail samples + numeric stats for JSON arrays, dedupe-with-counts plus guaranteed error-line survival for logs, head/tail windowing for text — with a deterministic stdlib script, no model call, no summarization risk. The information that defines meaning survives; the repetition that defines cost doesn't.
What This Skill Produces
- The crushed artifact — the compressed version, with its token math in the header (~6,000 → ~130 is typical for uniform JSON)
- The crush decision — what to crush, what to keep raw, and what to link instead of load
- The escape hatch — every crush names how to fetch the original when a detail turns out to matter
- The pipeline habit — where in the agent's workflow the crush step belongs (between tool and context, always)
Required Inputs
Ask for these if not provided:
- The payload — the JSON/log/text (or its path), and roughly how it will be used ("I need the error" vs. "I need every row" are opposite answers)
- The repetition question — is this data uniform (crushable to schema+stats) or is each row genuinely distinct (crushing loses signal — keep or filter instead)?
- The journey stage — one-shot analysis (crush hard) vs. data the conversation will keep querying (crush to an index, keep the original fetchable)
Programmatic Helper
python3 scripts/context_crush.py --mode json --file response.json
python3 scripts/context_crush.py --mode log --file build.log --keep 40
cat data.json | python3 scripts/context_crush.py --mode json
Deterministic, stdlib-only, no API. JSON arrays → {count, schema, head samples, tail, numeric min/max/mean} · logs → consecutive-duplicate collapse + first-occurrence dedupe + an always-preserved error/warning section · text → whitespace normalization + head/tail window with an elision marker. Inputs too small to gain are returned unchanged with an honest header.
Framework: The Crush Rules
- Crush between the tool and the context, not after: the token is spent the moment raw output enters the window — the crush step lives in the pipeline (
tool | crush | context), not in cleanup. Retroactive crushing saves nothing already paid for. - Structural beats semantic for data: summarizing JSON with a model costs tokens, adds latency, and can hallucinate; schema+samples+stats is free, instant, and provably faithful — the numbers are computed, not paraphrased. Save model-summarization for prose, where structure can't do the work.
- Errors are sacred: the log crusher's contract is that every error/warning line survives regardless of compression — a crush that can lose the one line that mattered is a corruption, not a compression. Any custom crushing keeps this invariant.
- The escape hatch is part of the artifact: every crushed block states where the original lives ("full response in /tmp/response.json — fetch rows by id if needed"), because reversibility is what makes aggressive crushing safe.
- Know when not to: non-uniform rows where each is signal, data being diffed byte-for-byte, legal/audit content, and anything under ~50 lines (the crush header costs more than it saves — the script says so itself). Crushing is a default for bulk, not a reflex for everything.
Output Format
Crushed: [payload] — ~[X] → ~[Y] tokens ([Z]% smaller)[The crushed artifact, script header included]
Kept raw: [what wasn't crushed and why] · Original: [where it lives, how to fetch] Pipeline note: [where the crush step now sits in this workflow]
Quality Checks
- The token math appears — before, after, percent
- JSON crushes carry schema and computed stats, never paraphrased numbers
- Every error/warning line in a log crush survived
- The original's location and fetch route are stated
- Too-small inputs were returned unchanged, honestly
Anti-Patterns
- Do not summarize data with a model when structure can compress it — paraphrased numbers are hallucination surface
- Do not crush non-uniform, every-row-is-signal data — filter or keep it
- Do not drop the escape hatch — irreversible compression turns a saving into a gamble
- Do not crush after the tokens are spent — the step belongs in the pipeline
- Do not let the crush eat errors — the invariant outranks the ratio
Based On
The context-compression layer pattern — structural compression of tool outputs before the LLM (as in Headroom) — rebuilt here as a keyless, deterministic, stdlib skill.