self-improving-loop

When to Use

Run this skill in the following situations.

When a performance regression of a specific agent/skill has been observed and root cause analysis and derivation of improvement measures are needed
When continuous improvement based on trace samples is needed in a weekly/biweekly regular quality review cadence
continuous-evalWhen this reported a 5%p drop in faithfulness or the negative user_feedback ratio exceeding the threshold
incident-responseWhen this classified the root cause of a SEV2/3 event as "prompt quality" or "missing tool invocation"

Do not use it in the following situations.

Weight improvement for closed managed models (Bedrock Claude/Nova, OpenAI GPT-4.1, Gemini 2.5, etc.) — weight access is unavailable. Only prompt/routing improvements are possible.
Automatic execution of the Train/Deploy stages — human approval is required according to the ADR, and this skill is only responsible up to proposals (PR drafts).

Prerequisites

Langfuse v3.x self-hosted or cloud instance (not provided by OMA — external prerequisite) AND The trace lookup MCP server must be registered in profile.yaml의 observability.trace_mcp. The skill calls it with the mcp__<server_name>__* (default mcp__langfuse__*) tool. If not configured, the skill exits with the message "trace MCP not configured".
Prometheus (AMP or self-hosted) — collects latency, error_rate, and token_usage metrics.
CloudWatch Logs — Agent execution logs, tool invocation logs.
awslabs.cloudwatch-mcp-server==0.0.25, awslabs.prometheus-mcp-server==0.2.15 A state in which the MCP server is configured (for supply chain security, a PyPI version pin is required instead of @latest ).
Write permission for the Git repository (for generating PR drafts).
The baseline evaluation score of the agent/skill to be improved must have been generated as continuous-evalwithin the last 24 hours.

4-Stage Loop

본 skill은 engineering-playbook의 self-improving-agent-loop.md In the 5-Stage design, Rollout·Score·Filter 3 stages are automated, and the 2 Train/Deploy stages are separated into PR approval and human trigger. In other words, the loop inside the skill is 4 stages.

Stage 1: Observe — Collect Langfuse traces

Group the recent traces of the agent/skill to be improved by time, quality, and cost dimensions.

Call the MCP tool mcp__langfuse__query_tracesto query traces (through the configured observability.trace_mcp server):

{
  "name": "$TARGET_AGENT",
  "from": "2026-05-27T00:00:00Z",
  "limit": 500
}

Save the output as .omao/plans/self-improving/traces-raw.jsonl.

The collected items are trace ID, prompt, response, reward_score, user_feedback, latency, tool_invocations, and token_count. Traces with low reward (< 0.5) and traces with negative user_feedback are flagged as priority analysis targets.

Stage 2: Analyze — Detect regression patterns

Analyze the collected traces along the three axes of quality, cost, and safety.

Quality axis: continuous-evalCompare the Ragas results (faithfulness, answer relevance, context precision) with the baseline for the same period. Classify metrics that dropped by 5%p or more as regressions.
Cost axis: Query rate(agent_token_total[1h])from Prometheus to check for surges in token usage.
Safety axis: Langfuse score 중 toxicity, pii_leakage Quarantine any trace labeled positive, even if it is a single instance.

The analysis results are saved as a report with the following structure.

{
  "target": "rag-qa-agent",
  "window": "2026-04-14T00:00:00Z/2026-04-21T00:00:00Z",
  "regressions": [
    {
      "dimension": "faithfulness",
      "baseline": 0.87,
      "current": 0.79,
      "delta_pp": -8.0,
      "affected_trace_ids": ["tr_abc", "tr_def", "..."],
      "hypothesized_cause": "system_prompt revision on 2026-04-16 removed citation instruction"
    }
  ]
}

Save path: .omao/plans/self-improving/report-${target}-${timestamp}.json

Stage 3: Propose — Generate Prompt/Skill revision proposals

Generate revision proposals that address the regression root cause hypotheses. All revision proposals are written in diff format so that a human can review them.

--- a/skills/rag-qa-agent/SKILL.md
+++ b/skills/rag-qa-agent/SKILL.md
@@ -42,7 +42,11 @@
 ## Instructions
 
 You are a RAG QA agent. Retrieve relevant context and answer the user's question.
-Always respond concisely.
+Always respond concisely and include inline citations in the form [source:doc-id]
+for every claim grounded in retrieved context. If no retrieved context supports
+a claim, state "I cannot verify this from available sources" instead of speculating.
+
+Never output PII tokens (email, SSN, phone) even if present in retrieved context.

Revision proposal generation principles:

Minimal change — propose the smallest change that addresses one regression root cause first.
Measurability — specify in the PR body the Ragas threshold (e.g., faithfulness ≥ 0.87) that the revision proposal must pass.
Rollback path — store the original before modification in .omao/plans/self-improving/rollback/ .
Human approval required — this skill only creates the PR as a draft and does not perform merges.

Stage 4: PR — Create Draft Pull Request

Create a PR draft with the GitHub CLI.

BRANCH="self-improving/${TARGET_AGENT}-$(date +%Y%m%d-%H%M)"
git checkout -b "$BRANCH"
git add skills/rag-qa-agent/SKILL.md
git commit -m "improve($TARGET_AGENT): restore citation instruction + block PII leakage"

gh pr create --draft \
  --title "improve($TARGET_AGENT): address faithfulness -8.0pp regression" \
  --body "$(cat <<'EOF'
## Self-Improving Loop Proposal

**Target**: `rag-qa-agent`
**Window**: 2026-04-14 .. 2026-04-21
**Detected Regression**: faithfulness 0.87 → 0.79 (-8.0pp)

### Evidence
- Affected traces: 47 traces, avg reward 0.42 (baseline 0.78)
- Langfuse query: see `.omao/plans/self-improving/report-rag-qa-agent-20260421.json`

### Hypothesized Cause
System prompt revision on 2026-04-16 removed the "include inline citations" instruction.

### Proposed Fix
Restore citation instruction + add explicit PII blocking clause.

### Acceptance Criteria
- [ ] `continuous-eval` faithfulness ≥ 0.87 on golden dataset
- [ ] No PII leakage detected in 48h canary (5% traffic)
- [ ] user_feedback positive rate ≥ baseline (0.71)

### Rollback
`.omao/plans/self-improving/rollback/rag-qa-agent-20260421.md` holds the pre-change content.

### ADR Compliance
Per ADR Self-Improving Loop §2, this proposal does NOT trigger model retraining. Merge of this PR results in prompt-only change; no weight update. `continuous-eval` will verify on next scheduled run.
EOF
)"

Example Inputs/Outputs

Input: self-improving-loop rag-qa-agent

Output:

.omao/plans/self-improving/traces-raw.jsonl — 500 traces from the last 7 days
.omao/plans/self-improving/report-rag-qa-agent-20260421.json — regression analysis
Draft PR: improve(rag-qa-agent): address faithfulness -8.0pp regression

Failure cases:

trace MCP (observability.trace_mcp) not set or inaccessible → .omao/state/self-improving/error-langfuse.logrecord the cause in , and exit the skill.
No regression detected → generate only a "no regression detected" report and do not create a PR.
If the target agent is not self-hosted but a managed model (Claude/Nova, etc.) → stop immediately due to ADR §1 violation and provide guidance: "managed model — prompt-only improvement required".

ADR boundary compliance checklist

Immediately before this skill finishes execution, automatically verify the following. If any one fails, stop PR creation.

Is the target a self-hosted model (Qwen3 / Llama 4 / GLM-5 family)?
Is the proposal limited to prompt/skill modifications (no weight update proposals)?
If there is a proposal to use training data, does it reference only traces that passed all 4 gates: PII / Consent / Region / Confidential?
Is the rollback path stored in .omao/plans/self-improving/rollback/?
Is the PR status draftand is there no auto-merge setting?

References

Official documentation

Langfuse OpenTelemetry — Production trace standard
Ragas Documentation — quality metric definitions
GitHub CLI gh pr create — PR draft creation reference

Technical blogs

LMSYS RouteLLM — regression analysis patterns
Langfuse — Production observability for LLM apps — cases of trace-based quality improvement

About this skill