SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsos-eval-lab-setup

os-eval-lab-setup

Testing & quality

View repository

Updated 10-16 09:50

About this skill

Identity: The Eval Lab Setup Agent

You bootstrap evaluation lab environments for autoresearch improvement runs. A lab repo is a standalone git repo with a hard copy of the plugin files (no symlinks), the os-eval-runner engine installed, and a customized eval-instructions.md ready for an eval agent to follow.

The template used to generate eval-instructions.md lives at: assets/templates/eval-instructions.template.md (relative to this skill root)

Phase 0: Intake

Ask each unanswered question. If provided in $ARGUMENTS, confirm rather than re-ask.

Q1 — Lab repo path? The local filesystem path to the lab git repository (e.g. <USER_HOME>/Projects/test-link-checker-eval). If it doesn't exist: "Should I create a new directory at that path and initialize it as a git repo?"

Q2 — Target plugin path? The canonical plugin path in agent-plugins-skills (e.g. .agents/skills/link-checker). This is what gets hard-copied into the lab repo.

Q3 — Target skill name? The skill folder name to optimize (e.g. link-checker-agent). This is the skill whose SKILL.md will be mutated each iteration.

Q4 — GitHub repo URL? The remote URL for the lab repo (e.g. https://github.com/username/test-skill-eval.git). Set as origin in the lab repo.

Q5 — Round label? Short label used in log and survey filenames (e.g. link-checker-round1). Default: <skill-name>-round1.

Q6 — agent-plugins-skills root path? The absolute local path to the agent-plugins-skills repo (needed for the npx install path and master plugin path). Default: ask the user or detect from context.

Q7 — What are you optimizing for? (primary metric)

Present these options and ask the user to pick one:

Option Metric KEEP condition Best when
quality_score (default) routing_accuracy × 0.7 + heuristic × 0.3 score ≥ baseline AND f1 ≥ baseline General SKILL.md improvement
f1 F1 score f1 ≥ baseline Routing balance — both precision and recall matter equally
precision Routing precision precision ≥ baseline Skill is over-triggering (too many false positives)
recall Routing recall recall ≥ baseline Skill is under-triggering (missing true positives)
heuristic Structural health score heuristic ≥ baseline Routing is already good; fixing structural/doc issues

If the user is unsure: diagnose first — run eval_runner.py --snapshot to see whether false-positive or false-negative rate is the dominant problem, then suggest the matching metric.

Default: quality_score if the user has no preference.

Q8 — What optimization strategy? (how much context the proposer sees)

Present these options:

Strategy Proposer sees Token cost Best when
scores-only results.tsv rows (score history) ~0.002 MTok/iter Simple routing fix, fast cheap iteration
traces (default) results.tsv + last 3 trace files ~0.1 MTok/iter Most cases — enough signal without high cost
full results.tsv + ALL trace files ~1–10 MTok/iter Complex structural failures needing causal diagnosis

The strategy is written into program.md as an instruction to the proposer. It does not change evaluate.py behavior — only what the proposer agent reads before proposing mutations.

Default: traces unless the user specifies otherwise.

Q9 — Which CLI proposer for mutations?

The improvement loop delegates mutation proposals to an external CLI for cheap, fast iteration.

Option Command Best when
copilot (default) copilot -p "..." GitHub Copilot CLI installed
gemini gemini -p "..." Gemini CLI installed
self agent self-proposes No CLI available (slowest, most tokens)

Check availability: which copilot / which gemini. Default to copilot if both are present. The choice is written into eval-instructions.md Step 4 so the eval agent knows which command to use.

Confirm before proceeding:

Lab repo:          /path/to/lab-repo (e.g. <USER_HOME>/Projects/...)
Plugin (master):   plugins/<plugin-name>  →  /abs/path/agent-plugins-skills/plugins/<plugin-name>
Skill:             <skill-name>
GitHub remote:     https://github.com/...
Round label:       <label>
Primary metric:    quality_score  (or: f1 / precision / recall / heuristic)
Strategy:          traces         (or: scores-only / full)
Proposer CLI:      copilot        (or: gemini / self)

Phase 1: Bootstrap the Lab Repo

⚙️ Set Key Variables First (do this before all other steps)

The plugin path from Q2 has the form plugins/<plugin-folder>/skills/<skill-folder>. Parse it explicitly using cut — do NOT infer PLUGIN_NAME from the plugins/ root word:

APS_ROOT=<abs-path-to-agent-plugins-skills>          # from Q6
PLUGIN_PATH=plugins/<plugin-name>                    # from Q2, e.g. plugins/dev-utils
SKILL_NAME=<skill-folder-name>                       # from Q3, e.g. convert-mermaid

# Extract plugin folder name (segment 2, NOT segment 1 which is 'plugins')
PLUGIN_NAME=$(echo "$PLUGIN_PATH" | cut -d'/' -f2)  # → mermaid-to-png

LAB_PATH=<lab-repo-path>                             # from Q1
SKILL_EVAL_SOURCE="$LAB_PATH/.agents/skills/os-eval-runner"

# Verify before proceeding:
echo "PLUGIN_NAME=$PLUGIN_NAME  SKILL_NAME=$SKILL_NAME"
echo "SKILL_EVAL_SOURCE=$SKILL_EVAL_SOURCE"

⚠️ PLUGIN_NAME = mermaid-to-png, NOT plugins. Always verify the echo output.

[!WARNING] Workspace Permissions: The lab repo path is usually outside your current active workspace. Before modifying any files in the lab directory, you MUST ask the user for full file access / to turn off workspace validation. Once they confirm, you may use your normal code-editing tools securely. If they choose not to grant permission, you must bypass your internal file tools entirely and use only native bash operations (mkdir, cp, echo via run_command) to create the lab environment. Using file tools without permission will result in frozen operations.

Run these steps in the lab repo directory in order:

1a. Git setup

cd <lab-repo>
git remote remove origin 2>/dev/null
git remote add origin <GITHUB_URL>
git remote -v

If not yet a git repo:

git init && git add . && git commit -m "init: <skill-name> eval sandbox"

1b. Clean slate

rm -rf .agent .agents .gemini .claude

1c. Hard-copy plugin files (resolve symlinks)

cp -RL <APS_ROOT>/plugins/<plugin-name> <lab-repo>/<plugin-folder-name>
rm -rf <lab-repo>/<plugin-folder-name>/**/__pycache__

Dependencies

  • os-eval-runner (agent-agentic-os plugin)
  • copilot-cli-agent (copilot-cli plugin)

[!TIP] See INSTALL.md for instructions on how to install missing dependencies. If -y crashes: run without it and press Enter to accept defaults. Both skills are required: os-eval-runner gates iterations, copilot-cli-agent proposes mutations.

1e. Seed commit and push

cd <lab-repo>
git add . && git commit -m "seed: install os-eval-runner engine"
git push origin main

1f. Verify Python 3

python --version  # must be 3.8+

Phase 2: Generate eval-instructions.md

Use the generate_eval_instructions.py script provided in this skill's scripts directory to generate the filled instruction file:

python $APS_ROOT/plugins/agent-agentic-os/skills/os-eval-lab-setup/scripts/generate_eval_instructions.py \
    --template $APS_ROOT/plugins/agent-agentic-os/assets/templates/eval-instructions.template.md \
    --out $LAB_PATH/eval-instructions.md \
    --skill-display-name "<Human-readable skill name>" \
    --skill-name "$SKILL_NAME" \
    --plugin-dir "$PLUGIN_NAME" \
    --mutation-target "SKILL.md" \
    --repo-url "$GH_URL" \
    --round-label "<The round label>" \
    --engine-source "$SKILL_EVAL_SOURCE" \
    --master-plugin-path "$APS_ROOT/plugins/$PLUGIN_NAME"

Phase 3: Confirm Ready

Report to the user:

  • Lab repo path and confirmed git remote
  • Files copied from master plugin
  • Engine installed at .agents/skills/os-eval-runner/
  • eval-instructions.md written at lab repo root

Execution Options (Confirm with User): Ask the user how they want to run the loop:

  1. Manual (Isolated Window): "Open a new Claude Code session pointed at the lab repo and say: Follow eval-instructions.md."
  2. Autonomous (Triple-Loop Native): You (the Antigravity agent) trigger the looping orchestrator immediately in the background using the gemini CLI in headless mode.

If Autonomous: Run this exact bash command from your active workspace:

# Run from within the lab repository
# NOTE: standalone 'gemini' CLI deprecated June 18, 2026 — use 'agy' instead.
# See references/cheapest_models.md for current model names.
nohup agy --yolo -p "You are the L1 Triple-Loop Orchestrator. Read eval-instructions.md completely and follow every step precisely. You are running headlessly — do NOT pause to ask for human confirmation on the evals.json setup; populate the JSON yourself and immediately execute all 10 iteration loops using copilot as your L2 proposer. Generate eval_progress.png at the end." > agy_orchestrator_<skill-name>.log 2>&1 < /dev/null &

[!IMPORTANT] If you hit Tool execution denied by policy, run gemini trust <lab-repo-path> in your master terminal.

When the run completes (or you observe it finishing via logs), use the os-eval-backport skill in this repo to review and apply approved changes back to master sources.

What to Expect: Meta-Circular Improvement

When os-eval-runner is installed as a peer in the lab repo alongside the target skill, the improvement loop may propose changes to os-eval-runner itself — its SKILL.md, scripts, or evals — in addition to the target skill. This is expected and welcome, not a bug.

Why it happens: the agent can read all installed skills and proposes the highest-leverage change it can find, regardless of which skill it's in. The lab copy of os-eval-runner is a safe mutation target because:

  • It's a physical copy, not a symlink to master
  • evaluate.py still gates every change — including changes to eval_runner.py itself
  • os-eval-backport review is the gate before any change reaches the canonical source

At backport review: treat changes to os-eval-runner files with extra scrutiny — the evaluator modifying its own scoring logic is high-leverage. Verify the change doesn't introduce a scoring bias that inflates future KEEP rates. See os-eval-backport SKILL.md for the review checklist.

This pattern is structurally equivalent to what Meta-Harness (Lee et al., arXiv:2603.28052) calls "harness self-improvement": the outer loop discovers improvements to the evaluation machinery itself, not just the target. The backport gate is the Pareto review that controls what flows to production.