SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillscaw-eval

caw-eval

Agents & models

View repository

Updated 10-15 20:13

About this skill

CAW Eval

End-to-end evaluation of CAW Agent quality: the local Mac acts as the scheduler, dispatching to remote servers to run headless claude (standard / recipe evaluation) or openclaw agent (weak model compatibility evaluation), with scoring and reporting all done locally.

Step 0: Environment identification (mandatory)

[[ "$(hostname)" == *openclaw* ]] && echo "env=openclaw" || echo "env=local"
  • env=local: continue. Ensure gcloud auth login is complete and the IAP channel is available.
  • env=openclaw: stop. This SKILL is alocal scheduler, and cannot be run directly on an openclaw server. Please return to the local Mac terminal and trigger it again.

Process routing

Evaluate three orthogonal dimensions:

Dimension flag Value
Evaluation mode --eval-mode e2e(default, full E2E including task_completion) / pact(evaluate only pact construction) / onboard(onboarding evaluation)
Recipe source --recipe-source real(default / call the real backend) / seed(recipe injected from the dataset) / empty(inject empty, control group)
Agent type Script split by file run_eval_cc.py（Claude Code headless）/ run_eval_openclaw.py(weak models such as doubao/minimax/gpt)

The old cli is still compatible:--eval-mode standard ↔ e2e、--eval-mode recipe ↔ pact；--recipe-mode cc_with_recipe/openclaw → seed、cc_no_recipe → empty、cc_real_recipe/oc_real_recipe → real。

Quick reference table (user intent → command template)

User says eval-mode recipe-source Script Server pool run_name template Detailed steps
"run evaluation" / "full-process evaluation" / default e2e real run_eval_cc.py SERVERS_CC_MAIN eval-cc-{model}-e2e-real-recipe-{TS} run-eval-cc.md
"real recipe" / "live recipe" + CC e2e or pact real run_eval_cc.py SERVERS_CC_MAIN eval-cc-{model}-{mode}-real-recipe-{TS} run-eval-recipe.md
"recipe evaluation" / "pact mode" + dataset recipe injection pact seed run_eval_cc.py SERVERS_CC_MAIN eval-cc-{model}-pact-seed-recipe-{TS} run-eval-recipe.md
"recipe control group" / "no recipe" pact empty run_eval_cc.py SERVERS_CC_CTRL eval-cc-{model}-pact-no-recipe-{TS} run-eval-recipe.md
"recipe comparison evaluation" Three runs:pact+seed、pact+empty、pact+real — use both scripts MAIN（seed/real）+ CTRL（empty） three runs run-eval-recipe.md
"weak model evaluation" / "doubao/minimax/gpt evaluation" / "openclaw evaluation" e2e or pact real or seed run_eval_openclaw.py SERVERS_{MODEL} eval-oc-{model}-{mode}-{source-alias}-{TS} run-eval-openclaw.md

run_name alias convention：real → real-recipe、seed → seed-recipe、empty → no-recipe. Template:eval-{cc|oc}-{model}-{eval-mode}-{source-alias}-{YYYYMMDD-HHMM}。

Default：run_eval_cc.py --eval-mode e2e --recipe-source real(i.e., sonnet headless full E2E + agent calling the real backend recipe).

Common prerequisites before execution (SSH / gcloud / server sync):common-execution.md

Overview

e2e / pact evaluation (CC headless)

The local Mac uses run_eval_cc.py dispatch to schedule N servers in parallel, each running headless claude -p。

本地 dispatch → 动态队列（空闲服务器自动取下一个 item）
               → 远端 claude headless 跑任务
               → scp 拉回 session → 上传 Langfuse → 评分 → 报告
  • Time: 17 cases / 3 machines ≈ 30-50 minutes (depending on difficulty)
  • Server pool(see run-eval-cc.md Step 2）：
    • SERVERS_CC_MAIN: 3 machines (test09-11), run e2e / pact + recipe-source=real|seed
    • SERVERS_CC_CTRL: 3 machines (test12-14), run only pact + recipe-source=empty control group
    • The two pools do not interfere with each other, so Recipe comparisons can run in parallel
  • Detailed steps:run-eval-cc.md
  • Comparison of five Recipe combinations:run-eval-recipe.md

Openclaw weak model evaluation

The local Mac uses run_eval_openclaw.py dispatch to schedule multiple openclaw servers in parallel, each running serially openclaw agent, session is uploaded directly to Langfuse, and a report is produced by local scoring.

本地 dispatch → 多台服务器 openclaw agent 跑任务 → 上传 Langfuse
             → 本地读 Langfuse trace 评分（LLM Judge subagent 并行）→ 报告
  • Time: 14 cases / 3 weak models ≈ 1-3 hours (depending on the model)
  • Detailed steps:run-eval-openclaw.md

Scoring System

e2e mode (full pipeline)

综合分 = task_completion × 0.3 + process_quality × 0.7
process_quality = S1(意图) × 0.15 + S2(Pact) × 0.45 + S3(执行) × 0.4

pact mode (only evaluates pact construction)

综合分 = S1(意图) × 0.20 + S2(Pact) × 0.45 + S3(交易构建) × 0.35
S3 = tx_construction_correctness × 0.5 + recipe_adherence × 0.3 + tx_submission_success × 0.2

No task_completion. Only evaluates whether the transaction is correctly constructed/submitted, not the on-chain execution result.

All scores are 0-1. See scoring.md。

Issue Attribution (used when writing reports)

Classify each finding into 7 layers: 🔵 SKILL under test / 🟢 Scoring System / 🟡 Dataset / 🟤 Recipe / 🟠 Evaluation Toolchain / 🔴 Product Code / 🟣 Runtime Environment. For details, see issue-attribution.md。

Dataset

Dataset Case Count Scenario Description
recipe-test-v3 7 uniswap-swap / aave-lend / weth-wrap Recipe evaluation (recommended), unified schema v2
standard-test-v3 7 Same as recipe-test-v3 Standard evaluation (recommended), same test set + different eval_mode for A/B
caw-agent-eval-seth-v2 14 transfer / swap / lend / dca / ... Old schema (pact_hints/stage_criteria), only for historical replay
caw-recipe-eval-seth-v1 - recipe multi-step Sepolia multi-step scenario, some items are partially in the new schema
  • Default recipe-test-v3(pact mode) / standard-test-v3(e2e mode)
  • recipe-test-v3 and standard-test-v3 contentcompletely identical(only metadata.eval_type differs), the difference is only at runtime:
    • --eval-mode pact --recipe-source seed: dispatch injects CAW_RECIPE_FILE, judge evaluates tx construction (not on-chain)
    • --eval-mode e2e --recipe-source real：dispatch does notinject CAW_RECIPE_FILE(agent autonomously caw recipe search), judge evaluates the full pipeline (including task_completion)
  • --dataset-name can specify other datasets
  • Dataset management:dataset-management.md
  • Dataset review (11 mechanical rules):dataset-review.md

Server Environment Setup

Create a new openclaw evaluation server (GCP instance / openclaw / caw / onboarding / top-up / verification): → server-setup.md

Scripts

Script Purpose
run_eval_cc.py CC evaluation orchestration (dispatch / run / upload / score / metrics)
run_eval_openclaw.py Openclaw evaluation orchestration (dispatch / run / upload / pack)
score_traces.py Scoring pipeline (assertions + judge → composite score → Langfuse)
judge_cc.py LLM-as-Judge (prompt construction)
assertions.py Structured extraction + threshold assertions
eval_utils.py Common utilities (Langfuse client / dataset / batch upload)
upload_session.py session → Langfuse trace
generate_dataset.py Dataset generation
validate_dataset.py Dataset schema validation
runtime_compliance.py Evaluation runtime compliance self-check
sync_to_servers.sh Server sync + hash verification