SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillscode-arena-cn

code-arena-cn

Coding & development

View repository

Updated 10-15 19:50

About this skill

CodeArena evaluation skill

Evaluate each project implementation bytwo independent rubrics(general Basic /100 + this round's requirement Round /100, each plus up to +10 dynamic bonus points), and produce abilingual (Chinese-English)report. This skill contains the complete standards, test cases, templates, and probe scripts.

0. Required resources (all within this skill)

  • references/EVAL_STANDARDS.md — authoritative standards (workflow, hard rules, baseline tool table).Read before evaluating.
  • references/basic_eval_cases.md — static Basic rubric (B1 Security 30 / B2 Visual 18 / B3 Code Quality 30 / B4 Governance 10 / B5 Test Coverage 12). Reused as-is each round.
  • references/report_template.md — report template (three-part structure, Chinese; English report mirrors it structurally).
  • references/scoring_overview.md — scoring overview.
  • references/example_round_eval_cases.md — one round of example test cases (a 0→1 requirement), for reference when generating new round test cases.
  • scripts/ — general libraries lib/(http/ui/port probing), the per-round generated templates/(api/ui/visual skeletons), the screenshot tool cap.mjs, the tool readiness check check-tools.sh and installation setup-tools.sh(see §4, §5).

1. Workflow (must be executed in order each round)

Paths are relative to the working root directory (both the evaluated projects and EvalSets/ are here);<round> is the short name of the round selected by the user.

  1. Tool readiness check (mandatory before starting each round) — first run bash scripts/check-tools.sh;Before using any tool, first confirm it is installed and available, install missing ones with bash scripts/check-tools.sh --install (installation methods for each item are in §4 and scripts/tools-README.md), and continue only after everything is ready.
  2. Select targets and name them — onetarget = (repository, branch); may span different repositories,and may also be multiple branches of the same repository(at least two targets). Projects are under the working root directory viarecursive discovery(may be in subdirectories), and may begiven by name only— names are assumed unique, resolved to paths using find <root> -type d -name <name> . For each repository recordthe resolved path + branch + report display name;Branch rules: no branch given → current branch; one/multiple branches given → each becomes a target, evaluated at itslast commit(report column name <repo>@<branch>). Record the mapping in EvalSets/<round>/<round>_targets.md(bilingual, including branches and resolved scope).
  3. Record the original prompt — verbatim into EvalSets/<round>/<round>_raw_prompt.md(bilingual).If the user did not give the evaluation projects/targets or did not give the original prompt, ask first — never guess targets or fabricate requirements.
  4. Execute(only after the user confirms 3.1):
    • 3.1 Generate from the prompt EvalSets/<round>/<round>_eval_cases.md(bilingual, /100 + bonus points) → submit to the user for review and update until confirmed.
    • 3.2 Pass the scope gate, start isolated temporary copies, and begin running;all evidence is written and retained EvalSets/<round>/evidence/。
    • 3.3 Run Basic test cases first (basic_eval_cases.md)。
    • 3.4 Then run Round test cases.
  5. Generate report — follow report_template.md to write EvalSets/<round>/<round>_test_report.md and _cn.md。

2. Hard rules (must be followed)

  • Scope = the set of files resolved for each target(run the scope gate first):default current branch—if there are uncommitted changes→evaluate the uncommitted delta (relative to HEAD), if clean→evaluatethe last commit content(git diff --name-only HEAD~1 HEAD, for a root commit take the full tree);explicitly specify branch—evaluate that branch'stip (last commit), ignore uncommitted changes in the working tree. For multiple branches in the same repo, use a separate temporary checkout for each (git worktree add <tmp-i> <branch> or git archive), without interfering with each other.Stagger ports for concurrent targets: prefer serial; if concurrent, target index i use an offset—backend PORT=<base>+i, frontend <base>+i, and have each frontend point to its own backend (BASE/FE_BASE/VITE_API_BASE or override the Vite proxy); only run concurrently when the backend URL is configurable.Must clean up after completion:pkill -9 node, restore the data directory, and for each checkout run git worktree remove <tmp-i> --force then git worktree prune, leaving no residue.
  • Each of the two rubrics is exactly 100 points, each with its own independent dynamic bonus (up to +10).
  • No double counting: the same issue/capability is counted only once. Cross-domain defects (such as hardcoded secrets) are deducted only in the most specific specialized item (→ B1-8), not deducted again in SAST; capabilities already scored (pagination/time formatting/status labels/filtering/error handling)are no longercounted as bonus.
  • Round isolation: each round is completely independent, only the current round is evaluated, and there is no comparison with or reference to historical rounds ("fixed/regressed/vs round X" is strictly prohibited). The only exception: B3d-2 compatibility may use git to compare with the previous version in the same repo (uncommitted vs HEAD, or HEAD vs HEAD~1), which is a technical diff within the item.
  • Independence: functional/security/UX scores come from probes developed by the evaluator; the project's own tests are counted only in the R4 test domain.
  • Bilingual artifacts: each deliverable produces English <name>.md + Chinese <name>_cn.md;sole exception evidence/(JSON/screenshots/scripts are not translated).
  • Report format: three-part format (summary / detailed scoring table / test case execution); the summary uses differentiated comparison—shared pros and cons are listed separately, differing items are presented in a table categorized by domain, bonus items are marked [bonus]; there is no "fix suggestions" section.

3. Baseline tools (fixed per domain, consistent across rounds)

Domain Tool
Functional correctness Backend: Node fetch probe (generated each round, see §5); frontend:playwright-cli skill for complete end-to-end UI testing + generated each roundUI test suite. Both use the generic library scripts/lib/{http,ui}.mjs, skeleton see scripts/templates/
Visual UX Playwright full-page screenshots at 3 viewports (generic engine scripts/cap.mjs; the screenshot checklist is calibrated each round according to the target page) + multimodal review
Code quality/security SAST Semgrep + Official semgrep-rules(each round from the official website git clone Dynamically fetched, not bundled/not frozen) + scripts/sast-rules.yaml(only as key/JWT supplementary rules)+ npm audit
Architecture Quality dependency-cruiser
Readability ESLint + eslint-plugin-complexity + jscpd (duplication rate must be ≤3%)
RESTful + Compatibility API probing + manual route review + git diff
Open Source Governance license-checker (only looks at dependencies, OSI permissive)+ npm audit --omit=dev(no High/Critical)
Test Coverage (Basic B5) c8(V8 coverage, wraps any runner)+ frontend vitest --coverage; map endpoints/roles/error paths/E2E write flows

4. Tool Readiness and Installation (be sure to install first)

Run the readiness check first, then install as needed —— Before using any tool, confirm it is available:

bash scripts/check-tools.sh            # 列出每个工具 OK / MISSING
bash scripts/check-tools.sh --install  # 自动安装缺失项,再复检

check-tools.sh Covers: semgrep, dependency-cruiser, eslint (+complexity), jscpd, license-checker, cloc, Playwright+Chromium, playwright-cli skill (and node/npm/git/python/pip). Per-item installation commands are below and in scripts/tools-README.md。

Choose one of the two environments (or use both):

A. One-click install on the host machine to ~/tools/ —— see scripts/setup-tools.sh(see details in scripts/tools-README.md):

chmod +x scripts/setup-tools.sh && ./scripts/setup-tools.sh   # 装 Semgrep 引擎 / dependency-cruiser / ESLint / jscpd / license-checker / cloc / Playwright
npx skills add https://github.com/microsoft/playwright-cli --skill playwright-cli   # 装 playwright-cli skill(完整 UI 测试)

Note: If downloading GitHub releases reports curl (16) HTTP2, the script already uses --http1.1 -C - to work around it; for Apple Silicon, the symlink directory uses /opt/homebrew/bin。

B. Self-provided inside the sandbox(evaluation actually runs here):

# Node 工具(本地装,避免污染全局)
npm i dependency-cruiser eslint@8 eslint-plugin-complexity jscpd license-checker
# Semgrep 引擎:先下 wheel 再 --no-deps 装,然后补依赖
pip install semgrep --break-system-packages         # 或下 wheel 后 pip install ./semgrep-*.whl

The Semgrep rule file must be dynamically fetched from the official website every round (not bundled, not cached or frozen):

git clone --depth 1 https://github.com/semgrep/semgrep-rules.git   # 每轮评测时现拉最新官方规则

Note: The sandbox proxy often returns 403 for GitHub release binaries, but git clone and pip/npm are available——so the official Semgrep rules always use git clone to fetch the latest version. This skill does not bundleofficial rule copies, and only bundles scripts/sast-rules.yaml as a key/JWT supplement for cases not covered by the official JS rule set. The current rubric does not include DAST, so OWASP ZAP / Dependency-Check are not installed; governance is handled by license-checker + npm audit.

5. Scripts: General Presets vs Per-Round Generation

Core principle: Test scripts (which determine "what to test"——assertions, flows, fields, test data) must be dynamically generated each round based on the current round's code + evaluation criteria + <round>_eval_cases.md use cases + <round>_raw_prompt.md requirements; only general infrastructure is pre-written.

General, preset (use directly, do not modify):

  • scripts/lib/http.mjs —— request/multipart/token extraction/result recorder (Recorder)。
  • scripts/lib/ui.mjs —— Playwright launch (including PW_EXEC/--no-sandbox), login, toast polling, console/page error capture, screenshots,UIRecorder。
  • scripts/lib/detect-ports.mjs / scripts/detect-ports.mjs —— detect the port from README/config.
  • scripts/cap.mjs —— 3-viewport screenshot tool (single login to avoid rate limiting).
  • scripts/sast-rules.yaml、setup-tools.sh、check-tools.sh、tools-README.md。

Generated per round (from the scripts/templates/ skeleton, do not run the template directly):

  • templates/api_test.template.mjs → generate EvalSets/<round>/evidence/harness/api_test.mjs: fill in the contract block and per-item assertions according to the target's real routes + use cases (each item corresponds to an eval-case ID).
  • templates/ui_test.template.mjs → generate …/harness/ui_test.mjs: fill in the complete write-operation flow according to the target frontend's routes/selectors/button text + use cases.
  • templates/visual_test.template.mjs → generate …/harness/visual_test.mjs: fill in which screens to capture according to the target page (for simple scenarios, directly use cap.mjs)。

Generation steps (workflow 3.x):

  1. Read <round>_eval_cases.md、<round>_raw_prompt.md, and review the target backend routes and frontend views/stores.
  2. Copy the corresponding template to EvalSets/<round>/evidence/harness/, and replace scripts/lib/ Copy the entire directory to …/harness/lib/(the template is referenced by ./lib/... , self-contained).
  3. Fill in the GENERATE-PER-ROUND section: contract, item-by-item assertions, UI flow, screenshot checklist—each item must be traceable to an eval-case; no generic placeholders allowed.
  4. Have the user confirm the generated harness, then run it.

Run (after generation):

# 端口先探测,不写死
node scripts/detect-ports.mjs <projectDir>
# 后端 API:
PROJECT_DIR=<projectDir> EVID_DIR=EvalSets/<round>/evidence \
  node EvalSets/<round>/evidence/harness/api_test.mjs <project> → <project>-api-results.json
# 完整 UI(必做):先用 playwright-cli skill 录制/核对选择器,再跑生成的 harness
PW_EXEC=<chromium> EVID_DIR=EvalSets/<round>/evidence/ui FE_BASE=http://localhost:<feport> \
  node EvalSets/<round>/evidence/harness/ui_test.mjs <project> → <project>-ui-results.json + 截图
# 视觉:
PROJECT_DIR=<projectDir> EVID_DIR=... node scripts/cap.mjs <project>
# SAST:
semgrep --config <semgrep-rules>/javascript --config scripts/sast-rules.yaml --json ...   # 甄别误报

Verifying only the API is not enough: frontend-specific defects (store recursion/corruption, API client miswired, dialog clicked but not submitted) are only visible with full UI tests; if a certain backend passes but the UI is broken, that API+UI item is judged partial at most, and the corresponding core scenario is judged Fail.

6. Key memory points (easy pitfalls)

  • The implemented interfaces/ports may differ greatly across rounds (multipart vs upload-then-JSON, PUT vs POST actions, accounts keyed by id vs username,/files whether auth is required, inconsistent ports)—so the harness is generated from the template each round(§5), first detect-ports.mjs detect the ports, then read the target routes and fill in assertions.Ports must always be detected from the README/code, never hardcoded(fall back to the default 3000/5173 only if detect-ports fails).
  • Clean the data directory thoroughly before starting the backend (some projects use a relative ./db, ignoring DB_DIR), otherwise 409 dirty data.
  • Repeated logins during visual evaluation trigger rate limiting → use a single login plus viewport-scaled screenshots.
  • API passing ≠ feature usable: be sure to use playwright-cli + the UI harness generated each round to run through all UI write operations—cases where the backend is correct but the frontend store/client is broken are very common (e.g., a Pinia local function with the same name as an import causing recursive stack overflow), and only UI tests can catch them.
  • Multiple branches: concurrent evaluations must stagger ports (target i offset by base+i, with each frontend pointing to its own backend); be sure to clean up after evaluation—pkill -9 node + each git worktree remove --force + git worktree prune, leaving no worktree/process/port behind.
  • Scoring must follow §2's "no double counting".