SuperAPI AI Directory
Partner中文AI newsFavorites
Skillsai-cli-benchmark

ai-cli-benchmark

Testing & quality

View repository

Updated 10-15 17:47

About this skill

AI CLI Benchmark

When to Use

  • Compare the speed, tokens, and cost of Copilot CLI, Codex CLI, and direct API calls.
  • Distinguish prompt cache reads from writes, and check whether cache-off can be claimed.
  • Build a reproducible benchmark with a fresh process, temporary home, and instruction/tool isolation in place.

If it is only a retrospective analysis of past sessions, it is out of scope. This skill handles the design, execution, and evidence-gathering of comparison conditions.

Cache Boundaries

  • Copilot CLI / Codex CLI have no public option to disable the provider-side model prompt cache. Even with a fresh process, temporary home, ephemeral, or nonce,cache offis not called that.
  • COPILOT_CACHE_HOMEis for local temporary data such as the Marketplace and auto-update packages.COPILOT_MCP_TOOL_CACHEis for the tool-list snapshot of a local MCP server. It is not a model prompt cache.
  • The OpenAI prompt cache cannot be manually cleared. Only when calling GPT-5.6 or later directly via the Responses API can you use explicit-only mode with no breakpoint to avoid cache writes for that request. Do not speculatively apply this to the CLI.
  • A nonce is an aid to avoid matching the entire user prompt; it does not prevent cache reads of the hidden system / agent prefix before it.

Workflow

  1. Freeze the task

    • Fix the dataset, expected labels, prompt, schema, input order, model, and reasoning, and take hashes.
    • When comparing accuracy and performance at the same time, apply the same pass criteria to all providers.

  2. Define measurement boundaries

    • one request、serial total、fresh-process end to enddistinguish between them.
    • Do not line up direct API calls and CLI startup-inclusive timemodel speedas the same thing.

  3. Isolate each CLI

    • Use an owned temp directory outside the repo, a fresh process, and a temporaryCOPILOT_HOME / auth-only CODEX_HOME.
    • Disable custom instructions, tools, MCP, and session reuse, and do not pass secrets to child processes or logs.

  4. Observe cache; do not assume control

    • If necessary, put a unique nonce at the beginning of the prompt, and save only the hash, not the nonce itself.
    • The zero-cache-read gate is for diagnosis. A failure is an observation of the provider cache, and should not be hidden as a benchmark failure.

  5. Record raw categories

    • Store normal input, cache read, cache write, output, and reasoning in separate fields.
    • Do not add token categories until you have confirmed the inclusion/exclusion relationships in the provider schema.

  6. Repeat before claiming performance

    • Do not attribute a single-run difference to a causal effect of the cache. Report median / p95 / range and accuracy.

  7. Validate the artifact

    python scripts/helper.py result.json `
  --require-cache-metadata `
  --report validation.json
    • Required shape: test_design.prompt_sha256, optional test_design.cache_observation (nonce_added_at_prompt_start, nonce_sha256), and providers[] with provider, elapsed_ms, passed, cases, usage (input_tokens, cached_input_tokens, cache_write_tokens, output_tokens, optional reasoning_tokens).
    • Add --require-zero-cache-read only for the diagnostic gate in step 4.

Reporting Rules

  • cache writeis not a cache hit. Separate it from reads, and calculate cost using the pricing category for the relevant model.
  • uncached runnot that, but as observed,cache read 0 runorcache-observed runwrite.
  • State the CLI version, model, reasoning, prompt hash, nonce hash, fresh-process conditions, measurement boundary, and usage schema.
  • Even with zero cache reads, this does not guarantee control over future writes or the hidden prefix.

Done Criteria

  • dataset/prompt/schema/model/reasoning are fixed, and there is a hash.
  • Use a fresh process and temporary home for all providers.
  • Store cache read/write in separate fields.
  • Report the measurement boundary together with accuracy.
  • Do not assign causality to speed differences without multiple runs.
  • benchmark JSON validator passes.

References