Mines Claude Code session JSONL for implicit user feedback. When a user corrects, redoes, reverts, or partially accepts AI output after a skill invocation, that signals a skill gap. Outputs structured feedback.jsonl with per-event dimension attribution for the improvement pipeline.
When to Use
- Compute per-skill correction rates to find which skills users correct most often.
- Generate
feedback.jsonlas input for improvement-generator's candidate prioritization. - Track correction trends over time (30-day rolling windows) to detect skill quality regression.
- Identify hotspot dimensions (accuracy vs coverage vs trigger_quality vs efficiency) per skill.
- Compare correction_rate before and after an improvement to validate whether a change actually helped.
- Audit a single skill's feedback history with
--skill-filterto understand why users reject its output. - Feed dimension hotspots into improvement-generator so candidates target the dimensions users care about.
- Bootstrap the auto-improvement loop: analyzer output is the starting signal that tells the pipeline which skills need work.
- Investigate spikes in correction_rate after a skill update to decide whether to rollback.
- Rebuild a noisy
feedback.jsonlwith--overwriteafter improving the analyzer rules. - Export markdown hotspot summaries for QMD or study-brain ingestion via
scripts/export_feedback_bridge.py.
When NOT to Use
- Synthetic task evaluation against a predefined task suite -- use improvement-evaluator instead.
- Structural scoring of SKILL.md quality (knowledge_density, coverage, completeness) -- use improvement-learner instead.
- Candidate multi-reviewer scoring with LLM judges -- use improvement-discriminator instead.
- Gate/accept decisions on improvement candidates -- use improvement-gate instead.
- Executing approved changes to skill files -- use improvement-executor instead.
- Generating improvement candidates from scratch -- use improvement-generator instead.
- Benchmark comparison against historical baselines -- use benchmark-store instead.
- Orchestrating the full generate-score-evaluate-execute-gate pipeline -- use improvement-orchestrator instead.
- Analyzing test runs or sub-agent sessions (these are filtered out automatically by
iter_session_files).
python3 scripts/analyze.py --session-dir ~/.claude/projects/ --output feedback-store/feedback.jsonl
python3 scripts/analyze.py --session-dir ~/.claude/projects/ --output feedback-store/feedback.jsonl --overwrite
python3 scripts/export_feedback_bridge.py --input feedback-store/feedback.jsonl --output feedback-store/$(date +%F)-session-feedback-hotspots.md
from scripts.metrics import load_feedback_events, compute_all_skill_metrics, format_metrics_report
events = load_feedback_events(Path("feedback-store/feedback.jsonl"))
report = format_metrics_report(compute_all_skill_metrics(events))
print(report)
# Skill Feedback Metrics
# ========================================
# cpp-expert: correction_rate=0.40 (n=20, corrections=6, partials=4, acceptances=10)
# hotspots: accuracy=5, coverage=3
# deslop: correction_rate=0.15 (n=40, corrections=4, partials=4, acceptances=32)
# hotspots: accuracy=3, efficiency=1
The output tells you cpp-expert has the highest correction rate (0.40) and its hotspot dimension is accuracy -- users most often correct naming/format issues. Feed this into improvement-generator with --source feedback-store/feedback.jsonl to generate candidates that prioritize accuracy fixes.
python3 scripts/analyze.py --skill-filter deslop --no-snippets --output feedback-store/deslop-feedback.jsonl
Then compute trend to check if recent changes improved the skill:
from scripts.metrics import load_feedback_events, compute_correction_trend
events = load_feedback_events(Path("feedback-store/deslop-feedback.jsonl"))
trend = compute_correction_trend(events, "deslop")
print(trend)
# {'skill_id': 'deslop', 'trend': -0.12, 'recent_rate': 0.10, 'prior_rate': 0.22,
# 'recent_sample': 18, 'prior_sample': 22, 'direction': 'improving'}
A negative trend (-0.12) with direction "improving" means the last 30 days had fewer corrections than the prior 30 days. The improvement worked.
# WRONG: session-feedback-analyzer does not execute tasks
python3 scripts/analyze.py --session-dir task_suite_results/ # meaningless
# RIGHT: use improvement-evaluator for synthetic task evaluation
python3 -m skills.improvement-evaluator.scripts.evaluate --task-suite task_suite.yaml
# WRONG: analyzer has no concept of SKILL.md structure
python3 scripts/analyze.py --session-dir ./skills/ # no JSONL files here
# RIGHT: use improvement-learner for SKILL.md structural scoring
python3 -m skills.improvement-learner.scripts.evaluate_skill --skill-dir skills/deslop/
Why Implicit Feedback Matters
Problem: improvement-evaluator uses a predefined task_suite.yaml to validate skill quality, but the task suite can only cover scenarios the author anticipated. Real users use skills in far more varied ways than the task suite -- they will use skills to do things the author never imagined. When users correct the AI output, this correction signal is implicit feedback pointing to the skill's shortcomings in real scenarios.
Because Implicit feedback comes from real usage rather than synthetic tests, and it can discover problems that a task suite can never find. For example, a certain skill's task suite has a 100% pass rate, but the user correction_rate is as high as 40% -- indicating that the task suite's test cases are severely disconnected from real needs. The output of session-feedback-analyzer (feedback.jsonl) is fed directly to improvement-generator, so that the generated candidates prioritize solving the dimensions users most frequently correct.
Tradeoff: The limitation of implicit feedback lies in signal noise. When a user says "wrong", it could be correcting the skill output, or it could be correcting their own previous instruction. The current keyword-based classifier cannot distinguish between these two cases, with a measured misjudgment rate of about 8%. The direction for improving accuracy is to introduce LLM-based classification (letting a small model determine what "wrong" refers to), but this introduces latency and cost. The current choice of keyword heuristic is because the 8% misjudgment rate is diluted under the statistical aggregation of correction_rate -- a misjudgment of a single event does not affect the overall trend judgment per-skill.
Why 3-Turn Influence Window
Tradeoff: A window that is too narrow (1 turn) will miss delayed corrections -- the user sees the AI output and continues to ask a question, only saying "that one just now was wrong" at turn 3. A window that is too wide (5+ turns) will introduce noise -- the user may already be discussing a completely different topic, and the "wrong" at this point is not a correction of the previous skill invocation. In measured tests, a 3-turn window has the best precision/recall balance: it captures 92% of real corrections, with a misjudgment rate of about 8%. In comparison, a 1-turn window captures only 71% of corrections, and a 5-turn window's misjudgment rate rises to 18%.
Because The window boundary is also constrained by next-invocation: if a new skill invocation appears within the 3-turn window, the current window is truncated at the new invocation. This prevents feedback for the second skill from being misattributed to the first skill. In the code, classify_outcome the next_invocation_idx parameter implements this truncation. The actual effect is that most windows have only 1-2 turns (users usually give feedback immediately), and scenarios that actually use the 3rd turn account for about 15%.
Priority rules within the window: When multiple turns within the window contain different signals (e.g., turn 1 says "okay" but turn 2 says "but the naming is wrong"), the classifier determines by the following priority: revert > redo > partial > correction > acceptance. This means that as long as any correction signal appears within the window, even if the first turn is acceptance, the final result is still correction or partial.
CLI
# Basic: analyze all sessions, write to default output
python3 scripts/analyze.py
# Custom session directory and output
python3 scripts/analyze.py --session-dir ~/.claude/projects/ --output feedback-store/feedback.jsonl
# Privacy mode: strip user message snippets
python3 scripts/analyze.py --no-snippets
# Filter to a single skill
python3 scripts/analyze.py --skill-filter cpp-expert
# Require at least 10 invocations before computing metrics
python3 scripts/analyze.py --min-invocations 10
# Combine flags: audit one skill privately with high threshold
python3 scripts/analyze.py --skill-filter deslop --no-snippets --min-invocations 10 --output feedback-store/deslop-audit.jsonl
# Rebuild after classifier fixes
python3 scripts/analyze.py --output feedback-store/feedback.jsonl --overwrite
| Param | Default | Description |
|---|---|---|
--session-dir |
~/.claude/projects/ |
Root directory containing session JSONL files |
--output |
feedback-store/feedback.jsonl |
Output path for the feedback JSONL file |
--no-snippets |
off | Omit user message snippets from output (privacy mode) |
--skill-filter |
none | Only analyze invocations of this specific skill |
--min-invocations |
5 | Minimum invocations before correction_rate is considered statistically meaningful |
--overwrite |
off | Rebuild the destination file instead of append+dedupe |
Detection Rules
Two ways to trigger skill detection:
Tool use detection: tool_use block，
name == "Skill"appears in the Assistant message, extract skill_id from input.skill .
This is the standard Claude Code skill invocation path.
Slash command detection: In the System message subtype == "local_command",
extract the skill name from the <command-name> tag.
Exclude built-in commands: help, clear, resume, compact, config.
| Path | Condition |
|---|---|
| Tool use | tool_use block, name == "Skill", skill_id from input.skill |
| Slash command | subtype == "local_command" + <command-name> tag |
Outcome Classification (3-turn influence window)
| Outcome | Type | Confidence | Trigger |
|---|---|---|---|
| correction | rejection | 0.9 | Keywords: "wrong", "incorrect", "no," (zh: "不对", "错了") |
| correction | revert | 0.9 | Git revert commands in assistant tool_use (git checkout/restore/reset) |
| correction | redo | 0.9 | Keywords: "try again", "redo" (zh: "重新来", "换个方案") |
| partial | partial | 0.7 | Qualifier ("but", "however", "但是") + correction or acceptance keyword |
| acceptance | explicit | 0.8 | Keywords: "lgtm", "looks good", "correct" (zh: "好", "可以", "对的") |
| acceptance | implicit | 0.6 | User message >20 chars, no question marks, no correction keywords |
Dimension Attribution
Each correction/partial gets a dimension_hint from keyword matching. When a user's correction message contains a specific keyword, that correction event is attributed to the corresponding evaluation dimension. This attribution result is passed to improvement-generator via feedback.jsonl, so that the candidates it generates prioritize the dimension the user most frequently corrects. If a keyword matches multiple dimensions, the one with the highest confidence is taken; if none match, it is marked as "unknown".
| Dimension | Keywords |
|---|---|
| accuracy | naming, format, style, typo, 命名, 格式, 拼写 |
| coverage | missing, forgot, incomplete, 缺少, 漏了 |
| reliability | again, inconsistent, 重复, 不稳定 |
| efficiency | slow, verbose, 太慢, 冗余 |
| security | security, secret, token, credential, 密钥 |
| trigger_quality | "wrong skill", "shouldn't trigger", "不该触发" -- wrong skill invoked entirely (distinct from accuracy which is correct skill, wrong output) |
correction_rate Formula
correction_rate = (corrections + 0.5 * partials) / total_invocations
partial is calculated with a weight of 0.5 -- partial acceptance means the skill output is partially correct,
lighter than a full correction, but still needs improvement.
When sample_size < --min-invocations(default 5), return sufficient_data: false,
to avoid statistical noise under small samples.
Trend calculation method: last 30d correction_rate vs prior 30d correction_rate. Positive delta = worsening (correction rate rising). Negative delta = improving (correction rate falling). |delta| <= 0.05 = stable (change is within statistical noise range). autoloop-controller uses trend to decide whether to continue iterating: if stable for two consecutive cycles, stop.
Output Artifacts
The primary output is feedback-store/feedback.jsonl -- one JSON object per line, one line per detected feedback event. Each event captures a single user reaction to a single skill invocation.
Schema (all fields present on every line):
{
"event_id": "a1b2c3d4...",
"timestamp": "2026-04-05T10:00:00Z",
"session_id": "uuid",
"skill_id": "cpp-expert",
"invocation_uuid": "msg-uuid",
"outcome": "correction",
"confidence": 0.9,
"correction_type": "rejection",
"user_message_snippet": "not right, should use const ref...",
"turns_to_feedback": 1,
"ai_tools_used": ["Read", "Edit"],
"dimension_hint": "accuracy"
}
| Field | Type | Description |
|---|---|---|
event_id |
string | SHA-256 hash prefix (16 chars) of invocation_id:skill_id, guarantees deduplication |
timestamp |
ISO 8601 | When the skill was invoked (not when the user responded) |
session_id |
string | JSONL filename stem, identifies the Claude Code session |
skill_id |
string | Which skill was invoked (e.g. "cpp-expert", "deslop") |
invocation_uuid |
string | UUID of the assistant message that triggered the skill |
outcome |
enum | One of "correction", "partial", "acceptance" |
confidence |
float | 0.6-0.9, how confident the classifier is in the outcome label |
correction_type |
string? | "rejection", "revert", "redo", "partial", or null for acceptances |
user_message_snippet |
string | First 200 chars of the user's response (empty when --no-snippets) |
turns_to_feedback |
int | How many user turns after invocation the feedback appeared (1-3) |
ai_tools_used |
string[] | Tools the assistant called between invocation and user response |
dimension_hint |
string? | Attributed evaluation dimension ("accuracy", "coverage", etc.) or null |
Secondary artifacts:
feedback-store/archive/feedback-YYYYMMDD.jsonl-- events older than 90 days, auto-archived byarchive_old_events().- Console summary printed to stdout after each run: event count, outcome distribution, top-5 skills by invocation count.
feedback-store/YYYY-MM-DD-session-feedback-hotspots.md-- markdown bridge output for QMD/study-brain ingestion, generated byscripts/export_feedback_bridge.py.
Metrics API (scripts/metrics.py)
from pathlib import Path
from scripts.metrics import (
load_feedback_events,
compute_correction_rate,
compute_correction_trend,
compute_hotspot_dimensions,
compute_all_skill_metrics,
format_metrics_report,
)
events = load_feedback_events(Path("feedback-store/feedback.jsonl"))
# Per-skill correction rate
compute_correction_rate(events, "cpp-expert")
# -> {"correction_rate": 0.35, "sample_size": 20, "sufficient_data": True,
# "corrections": 5, "partials": 4, "acceptances": 11}
# Trend over rolling 30-day windows
compute_correction_trend(events, "cpp-expert")
# -> {"trend": -0.08, "direction": "improving", "recent_rate": 0.30,
# "prior_rate": 0.38, "recent_sample": 12, "prior_sample": 8}
# Dimension hotspots (which dimensions get corrected most)
compute_hotspot_dimensions(events, "cpp-expert")
# -> {"accuracy": 5, "coverage": 3}
# All skills at once
all_metrics = compute_all_skill_metrics(events)
print(format_metrics_report(all_metrics))
Privacy Controls
--no-snippets strips user message snippets from feedback.jsonl output。
~/.claude/feedback-config.json with {"enabled": false} disables all collection.
analyze.py checks this configuration at startup; if disabled, it exits directly.
Directories automatically skipped:
pytest/-- sessions produced by tests are not real user behavior/tmp/-- temporary sessions/subagents/-- sub-agent sessions do not reflect direct user intentobserver-sessions/-- memory observer / synthetic relay sessions, which should not pollute real skill feedback
Auto-archives events >90 days old to feedback-store/archive/
to keep the active feedback store small and fast to query。
Slash command detection also validates that the command maps to a real installed skill. Builtins such as /read and /exit are ignored instead of being misattributed as skill invocations.
Related Skills
| Skill | Relationship | Data Flow |
|---|---|---|
| improvement-generator | Primary consumer | Reads feedback.jsonl via --source; uses dimension hotspots to prioritize candidates |
| improvement-evaluator | Complementary | Synthetic evaluation (task_suite.yaml) covers designed scenarios; analyzer covers real usage |
| improvement-learner | Orthogonal | Learner scores SKILL.md document structure; analyzer scores user interactions at runtime |
| improvement-discriminator | Downstream | Discriminator scores candidates that were generated based on analyzer's feedback signals |
| autoloop-controller | Control loop | Uses correction_rate trend for plateau/convergence detection; stable trend = stop iterating |
| improvement-gate | Downstream | Gate validates changes; analyzer provides the "before" baseline that gate compares against |
| benchmark-store | Historical | Stores correction_rate snapshots for long-term Pareto front tracking |
Generator auto-discovers feedback-store/ via lib/common.py:load_source_paths(). Hotspots inform prioritization -- when the correction count of a certain dimension is significantly higher than other dimensions, the generator prioritizes generating candidates for that dimension. autoloop-controller uses correction_rate plateau as termination condition: if the trend is stable for two consecutive 30d windows, convergence is determined.
Scripts & Tests Reference
| File | Purpose |
|---|---|
scripts/analyze.py |
Main analyzer: parses sessions, classifies outcomes, writes feedback.jsonl |
scripts/export_feedback_bridge.py |
Convert feedback.jsonl into markdown hotspot lessons for QMD/study-brain |
scripts/metrics.py |
Metrics library: correction_rate, trend, hotspots, report formatting |
tests/test_analyze.py |
Invocation detection, observer filtering, overwrite rebuild, and classification coverage |
tests/test_metrics.py |
11 test cases covering rate computation, trend direction, hotspot grouping, report format |
Run all tests:
cd skills/session-feedback-analyzer && python3 -m pytest tests/ -v