Skill: orca-eval
WHAT
Purpose and context
- Purpose: Process self-improvement loop: re-read session logs → extract best practices / anti-patterns → md report → propose actions (promote to skill, edit CLAUDE.md, add hook…). The report is a proposal — every action must go through
propose → gate, never performed automatically.
- Trigger (when to use):
/orca-eval [N] — N = number of most recent sessions to scan (e.g. 5/10/15)- No N → scan only the current session
- User says "evaluate the session", "extract best practices", "optimize the process from history"
- Non-goals: does not perform actions itself (promote skill, edit CLAUDE.md, add hook); does not modify/delete session logs; does not auto-trigger every 30 sessions (Phase 2, not part of this skill).
Mental model
*.jsonl session logs → orca-eval-scan.sh digest (prompts · tool errors · lệnh lặp) → 4 tín hiệu (Correction · Repetition · Friction · Win) → finding → đúng 1 action (promote-to-skill · update-CLAUDE.md · add-hook · keep · ignore) → report draft → DỪNG chờ duyệt.
Input and output contract
|
Field |
Required? |
Meaning |
| In |
N |
no |
number of most recent sessions by mtime *.jsonl; default = current session |
| In |
session logs ~/.claude/projects/<project-slug>/*.jsonl |
yes |
read-only |
| Out |
llmwiki/wiki/draft/orca/DDMMYY-eval-report.md |
yes |
according to the Report template |
| Out |
wiki/index.md + wiki/log.md updated |
yes |
— |
| Out |
action table shown to the user |
yes |
"done" = the user has seen the table; the action has not been executed |
Rules and capabilities
- RULE-01 (MUST): Read-only with session logs — do not modify/delete
*.jsonl.
- RULE-02 (MUST): Do NOT load the entire transcript — 1 session can be hundreds of KB; read via the scanner's digest.
- RULE-03 (MUST): Each finding is tied to exactly 1 action; the action is performed only after the user approves (via
propose if it is a code/skill change).
- RULE-04 (MUST): The Claude Code JSONL format has no public spec — if the scanner fails, fall back to the current session and state it clearly; do not produce an empty report.
- Capabilities: read local session logs via the scanner; write draft wiki + index + log. No network, no modifying code/config.
Failure boundaries
- Scanner fails → partial: fall back to the current session, state it clearly; do not produce an empty report.
- No signal at all → the report states 0 findings explicitly (do not fabricate findings).
- User has not approved → the action stays in proposedstate, the skill stops.
HOW
Main workflow
| Step |
Type |
Inputs |
Action |
Outputs/exit |
Failure/next |
| W01 |
deterministic |
N |
Scan: skills/orca-eval/assets/orca-eval-scan.sh [N] from repo root |
digest |
fail → B01 |
| W02 |
judgment |
digest |
Distill 4 types of signals |
list of findings + evidence |
— |
| W03 |
effect |
finding |
Report per template + update index/log |
draft report |
— |
| W04 |
judgment |
finding |
Action proposal: action table for user, STOP |
action table |
not approved → stop |
Details of each step (source of truth for W01–W04):
- Scan — run
skills/orca-eval/assets/orca-eval-scan.sh [N] from repo root. The script returns a compact digest (user prompts, tool errors, repeated Bash commands). Do NOT load the entire transcript — 1 session can be hundreds of KB.
- Distill — from the digest, find 4 types of signals:
- Correction: user corrects the agent's words ("no, use X", "wrong", "actually…") → new behavior rule
- Repetition: sequence of commands/operations repeated ≥ 3 times across sessions → skill-ification candidate
- Friction: repeated tool errors, multiple permission prompts, retry loops → hook/permission/config candidate
- Win: workflow the user approves quickly, without needing edits → record as best practice to keep as is
- Report — write
llmwiki/wiki/draft/orca/DDMMYY-eval-report.md per the template below. Update wiki/index.md + wiki/log.md.
- Action proposal — each finding is attached to exactly 1 action:
promote-to-skill / update-CLAUDE.md / add-hook / keep / ignore. Display the action table to the user. STOP — actions are only performed after the user approves (via propose if it is a code/skill change).
Input
| Parameter |
Meaning |
Default |
N |
Number of most recent sessions (by mtime of *.jsonl) |
current session |
Session logs are located at ~/.claude/projects/<project-slug>/*.jsonl — slug is the cwd with / replaced by - (e.g. -Users-giatran-orca-workspaces-setup-evaluation).
Branches
| ID |
Kind |
Guard |
Behavior |
Skip / failure |
Rejoin |
| B01 |
recovery |
scanner fail (JSONL format changed) |
fallback to current session, report clearly in the report |
nothing to scan → report, do not generate an empty report |
W02 |
| B02 |
conditional_required |
action is a code/skill change and the user approves |
switch to propose → gate |
user does not approve → stop |
— |
Validation and stopping
The scanner is deterministic; signal classification is judgment — each finding must have evidence (session + short quote) in the table. Hard stop at W04 awaiting user approval.
Examples
- Positive:
/orca-eval 10 → scan the latest 10 *.jsonl → detect the user 4 times editing "use ./scratchpad/ not /private/tmp" (Correction) → finding tagged update-CLAUDE.md; report llmwiki/wiki/draft/orca/DDMMYY-eval-report.md + action table, STOP.
- Boundary/failure:
orca-eval-scan.sh fails due to a JSONL format change → fall back to the current session, report clearly states the scanner failed; if there is no signal → report 0 findings, do not generate an empty report.
Reference — Report template
---
type: draft
title: "DDMMYY-eval-report"
status: proposed
tags: [<skill-name>, output-report]
timestamp: YYYY-MM-DD
---
# DDMMYY-eval-report
**Type:** draft
**Status:** proposed
**Tags:** orca-eval, eval-report
**Proposed:** YYYY-MM-DD
**Scope:** <N session / session hiện tại> — <danh sách file jsonl đã quét>
## Best practices
| # | Tín hiệu | Loại | Bằng chứng (session, trích dẫn ngắn) |
|---|----------|------|--------------------------------------|
## Đề xuất action
| # | Finding | Action | Lý do |
|---|---------|--------|-------|
## Origin
- **Sessions:** <paths đã quét>
- **Generated by:** /orca-eval
Reference — Limits
- Read-only with session logs — do not modify/delete
*.jsonl.
- Claude Code's JSONL format has no public spec — if the scanner fails, fall back to the current session and report clearly, do not generate an empty report.
- Auto-trigger every 30 sessions: Phase 2 (needs Stop-hook + counter), not yet part of this skill.
Reference — References (community — claimed 2026-06-11)
Origin
- Raw:
llmwiki/raw/evalution-engineering.md
- Draft:
wiki/draft/orca/110626-orca-eval-skill.md
