rock-agent-debug

Rock Agent Debug Skill

Via rockcli query and troubleshoot Agent Jobs (Harbor Job / Bash Job) running in the ROCK sandbox, and ultimately produce a detailed report saved to disk.

Background

The ROCK sandbox supports two Job types:

Type	Config class	Trial class	Identifying features
Harbor Job	`HarborJobConfig`	`HarborTrial`	Has `orchestrator`、`datasets`、`agents`、`jobs_dir`、`experiment_id` fields
Bash Job	`BashJobConfig`	`BashTrial`	Has `script` or `script_path` fields, no harbor-related fields

Job type is automatically detected via YAML configuration: first try HarborJobConfig parsing, and if it fails, fall back to BashJobConfig。

Important principles

Stop querying once you have enough information. Do not fall into the loop of "if the file is not in the expected location, search everywhere." Criteria:

Harbor Job（/data/logs/user-defined/）：.out The file contains the complete run log, and seeing the reward is enough
Harbor Job（/tmp/harbor/jobs/）：result.json + trial.log + exception.txt is enough
Bash Job：.out Logs + script exit code + artifact (if any) is enough
If the key files can explain the situation,no need to keep looking, and directly produce the report using the existing information

Step 1: Confirm the sandbox ID and Job type

# 查看沙箱存活状态
rockcli sandbox <id> status

# 同时检查所有可能的路径
rockcli sandbox <id> exec 'echo "=== /tmp/harbor/jobs/ ==="; ls -lt /tmp/harbor/jobs/ 2>/dev/null | head -10; echo "=== /data/logs/user-defined/ ==="; ls -lt /data/logs/user-defined/ 2>/dev/null | head -10; echo "=== /data/logs/ ==="; ls -lt /data/logs/*.out /data/logs/*.log 2>/dev/null | head -10'

Determine the Job type:

Read the YAML configuration (preferred):rockcli sandbox <id> exec 'cat /data/logs/user-defined/<job_name>.yaml'
If the configuration contains orchestrator、datasets、agents → Harbor Job
If the configuration contains script or script_path → Bash Job
If there is no YAML, only /tmp/harbor/jobs/ files under → Harbor Job(local harbor run)

If the user did not provide a sandbox ID, ask directly.

Step 2: Locate the target Job

When the user has not specified a job, take the most recent one with content (skip empty directories).

# Harbor 路径（本地 harbor run）：找最新非空的 job
rockcli sandbox <id> exec 'for d in $(ls -t /tmp/harbor/jobs/); do if [ "$(ls /tmp/harbor/jobs/$d 2>/dev/null)" ]; then echo $d; break; fi; done'

# ROCK 平台路径：列出 .yaml 配置和 .out 日志
rockcli sandbox <id> exec 'ls -lt /data/logs/user-defined/*.yaml /data/logs/user-defined/*.out 2>/dev/null | head -10'

Step 3: Read Job information

Harbor Job

/tmp/harbor/jobs/ path (local harbor run)

rockcli sandbox <id> exec 'cat /tmp/harbor/jobs/<job_name>/config.json'
rockcli sandbox <id> exec 'cat /tmp/harbor/jobs/<job_name>/result.json 2>/dev/null || echo "no result yet"'
rockcli sandbox <id> exec 'cat /tmp/harbor/jobs/<job_name>/job.log 2>/dev/null'

/data/logs/user-defined/ path (ROCK platform)

# 配置（agent、model、dataset）
rockcli sandbox <id> exec 'cat /data/logs/user-defined/<job_name>.yaml'

# 完整运行日志（含进度、reward、时间线）
rockcli sandbox <id> exec 'cat /data/logs/user-defined/<job_name>.out'

# result.json（可能不存在，不要为此继续搜索）
rockcli sandbox <id> exec 'cat /data/logs/user-defined/jobs/<job_name>/result.json 2>/dev/null || echo "no result.json"'

.out The file is sufficient to determine whether the task is complete(including the reward line, completion time, trial name); do not, just because result.json does not exist, search everywhere.

Bash Job

# 配置（script / script_path）
rockcli sandbox <id> exec 'cat /data/logs/user-defined/<job_name>.yaml'

# 完整运行日志
rockcli sandbox <id> exec 'cat /data/logs/user-defined/<job_name>.out'

# 检查是否有 artifact 输出
rockcli sandbox <id> exec 'ls -lt /data/logs/artifacts/ 2>/dev/null | head -10'

Characteristics of Bash Job:

Single script execution, no concept of trials
The exit code is the key to determining success/failure
There may be OSS artifact output to artifacts/{namespace}/{experiment_id}/{job_name}/
Logs are output directly to .out file

Step 4: Analyze results

Harbor Job: analyze trial by trial

Trial file structure (/tmp/harbor/jobs/ path):

<trial_name>/
├── result.json          # 完整结果（阶段时间、reward、异常）
├── trial.log            # 简短摘要（失败时有错误原因）
├── exception.txt        # 完整异常栈
├── agent/setup/stdout.txt   # Agent 安装日志
├── agent/setup/return-code.txt
├── verifier/reward.txt
├── verifier/test-stdout.txt
├── verifier/test-stderr.txt
└── agent/trajectory.json    # Agent 执行轨迹（ATIF 格式，若有）

# 列出 trials
rockcli sandbox <id> exec 'ls /tmp/harbor/jobs/<job_name>/ 2>/dev/null || ls /data/logs/user-defined/jobs/<job_name>/ 2>/dev/null'

# 批量查看所有 trial 状态（快速）
rockcli sandbox <id> exec 'for t in /tmp/harbor/jobs/<job_name>/*/; do echo "=== $(basename $t) ==="; cat "$t/result.json" 2>/dev/null | python3 -c "import json,sys; d=json.load(sys.stdin); print(\"reward:\", d.get(\"verifier_result\"), \"exception:\", bool(d.get(\"exception_info\")))" 2>/dev/null || cat "$t/trial.log" 2>/dev/null | head -3; done'

Description of key fields in result.json:

Field	Meaning	Anomaly signal
`exception_info`	Exception type and message	Non-null indicates an uncaught exception
`verifier_result.rewards`	Scores for each item	All 0 or null indicates the verifier did not run
`agent_result`	Agent execution result	null indicates the agent did not complete normally
`agent_execution.duration_sec`	Agent runtime duration	Extremely short (<5s) indicates early exit
`environment_setup`	Environment startup time	Extremely long indicates an image build or startup issue
`started_at` / `finished_at`	Start/End time	finished_at being null indicates it is not finished

Determine the current stage of the Trial (from result.json):

environment_setup has a value,agent_setup is null → environment is starting
agent_setup has a value,agent_execution is null → Agent is installing
agent_execution has a value,verifier is null → Agent is executing
verifier has a value,finished_at is null → Verifier is running
finished_at has a value → completed

Bash Job: single execution analysis

Bash Job has no concept of trial; it is a one-off script execution:

# 查看退出码（.out 末尾通常有 exit code）
rockcli sandbox <id> exec 'tail -20 /data/logs/user-defined/<job_name>.out'

# 搜索错误
rockcli sandbox <id> exec 'grep -i "error\|exception\|failed\|traceback" /data/logs/user-defined/<job_name>.out | tail -30'

# 如果有 artifact，检查输出
rockcli sandbox <id> exec 'ls -R /data/logs/artifacts/<namespace>/<experiment_id>/<job_name>/ 2>/dev/null'

Step 5: In-depth debugging (when problems are found)

Harbor Job debugging

Read detailed logs

# 1. 如果有 exception.txt，先读它
rockcli sandbox <id> exec 'cat /tmp/harbor/jobs/<job_name>/<trial_name>/exception.txt'

# 2. Verifier 输出（测试脚本的实际运行结果）
rockcli sandbox <id> exec 'cat /tmp/harbor/jobs/<job_name>/<trial_name>/verifier/test-stdout.txt'
rockcli sandbox <id> exec 'cat /tmp/harbor/jobs/<job_name>/<trial_name>/verifier/test-stderr.txt'

# 3. Trial 主日志（先搜关键词）
rockcli sandbox <id> exec 'grep -i "error\|exception\|failed\|traceback" /tmp/harbor/jobs/<job_name>/<trial_name>/trial.log | tail -50'

Check Docker Container status

Harbor Docker container naming rules:hb__<environment_name>(lowercase)

# 列出所有 Harbor 相关 container
rockcli sandbox <id> exec 'docker ps -a | grep "hb__"'

# 查看 container 详情
rockcli sandbox <id> exec 'docker inspect <container_name> | python3 -c "
import json, sys
data = json.load(sys.stdin)[0]
state = data[\"State\"]
print(f\"Status: {state[chr(39)]Status{chr(39)}}\")
print(f\"ExitCode: {state[chr(39)]ExitCode{chr(39)}}\")
print(f\"OOMKilled: {state[chr(39)]OOMKilled{chr(39)}}\")
"'

# 查看 container 最近日志
rockcli sandbox <id> exec 'docker logs <container_name> --tail 100'

View Agent execution trace (as needed)

Only check when the user wants to know what the agent did.

rockcli sandbox <id> exec 'cat /path/to/trial/agent/trajectory.json | python3 -c "
import json, sys
d = json.load(sys.stdin)
steps = d.get(\"steps\", [])
print(f\"共 {len(steps)} 步\")
for i, s in enumerate(steps[:20]):
    tool = s.get(\"tool\", \"\") or s.get(\"action\", \"\")
    summary = s.get(\"observation\", \"\") or s.get(\"result\", \"\")
    print(f\"  [{i+1}] {tool}: {str(summary)[:100]}\")
"'

Bash Job debugging

# 1. 查看完整日志
rockcli sandbox <id> exec 'cat /data/logs/user-defined/<job_name>.out'

# 2. 检查沙箱内执行环境
rockcli sandbox <id> exec 'echo "=== Python ==="; python3 --version 2>/dev/null; echo "=== Node ==="; node --version 2>/dev/null; echo "=== Disk ==="; df -h / 2>/dev/null'

# 3. 如果有 artifact，查看内容
rockcli sandbox <id> exec 'cat /data/logs/artifacts/<namespace>/<experiment_id>/<job_name>/* 2>/dev/null | head -100'

Step 6: Generate report (must output and save to a local file)

At the end of each analysis, you must:

Output a structured report in the conversation
Save the report to a local file:/tmp/rock-agent-report-<sandbox_id_短>-<timestamp>.md

Harbor Job report format

## Harbor Job 报告

**沙箱**: <sandbox_id>
**Job**: <job_name>
**类型**: Harbor Job
**时间**: <started_at> → <finished_at 或 "运行中">
**耗时**: <duration>

### 配置
- Agent: <agent_name> (<model_name>)
- Bench: <dataset_name> v<version>（或 Task: <task_path>）
- 并发数: <n_concurrent_trials>
- Verifier 模式: <harbor/native>

### 整体结果
- 总 Trials: <n>，成功: <n>，失败: <n>
- 平均 Reward: <mean>

### Trial 明细
| Trial | 状态 | Reward | 耗时 | 问题 |
|-------|------|--------|------|------|
| <name> | ✅/❌/🔄/⚠️ | <reward> | <duration> | <issue 或 -> |

### 执行时间线
- `<HH:MM>` 环境启动完成（耗时 Xs）
- `<HH:MM>` Agent 安装完成（耗时 Xs）
- `<HH:MM>` Agent 开始执行
- `<HH:MM>` Verifier 完成，reward=<N>

### 问题分析（如有）

**[CRITICAL] 问题标题**
- 影响 trial：`<trial_name>`
- 根本原因：...
- 证据：`exception.txt` 关键日志
- 建议：...

### 行动建议
1. **立即修复**：...
2. **下次运行前**：...

Bash Job report format

## Bash Job 报告

**沙箱**: <sandbox_id>
**Job**: <job_name>
**类型**: Bash Job
**时间**: <started_at> → <finished_at>
**耗时**: <duration>
**退出码**: <exit_code>

### 配置
- Script: <script 内容或 script_path>
- Environment: <environment 配置>
- Timeout: <timeout>

### 执行结果
- 状态: ✅ 成功 / ❌ 失败
- 退出码: <0 或非零>
- 关键输出: <摘要>

### 日志摘要
<关键日志片段，特别是错误信息>

### 问题分析（如有）
- 根本原因：...
- 证据：日志关键行
- 建议：...

### 行动建议
1. ...

Status icons: ✅ Passed | ❌ Failed | 🔄 Running | ⚠️ Completed but abnormal

Common failure quick reference

Harbor Job

Symptom	Failure stage	Typical cause
"Docker compose command failed" / "no such host"	env_setup	Image pull DNS failure
`agent/setup/return-code.txt` = 100	agent_setup	Agent installation script failure
`AgentTimeoutError`	agent_execution	Agent timeout
reward=0, no abnormality	verifier	Agent completed but did not solve the problem
`RewardFileNotFoundError`	verifier	reward.txt does not exist
`no running event loop`	cleanup	Cleanup phase crashed after Agent timeout (not the main failure cause)
reward.txt is empty or does not exist	verifier	verifier did not run to the step of writing reward
agent_result is null, execution time is extremely short	agent_setup	agent installation script error
All trials failed with the same exception	Environment level	Image/dependency/network issues

Bash Job

Symptom	Typical cause
Exit code non-zero	Script execution failed
Timeout exit	`timeout` Set too short or script hung
Log truncated	Insufficient sandbox resources (OOM / disk full)
No artifact output	Script did not write to the expected path or path permission issue

Agent type reference

Agent name	Description
`claude-code`	Claude Code CLI
`terminus` / `terminus-2`	Terminus internal agent
`swe-agent-internal`	Internal SWE-Agent implementation
`openhands`	OpenHands agent
`mini-swe-agent`	Lightweight SWE-Agent
`aider`	Aider coding agent
`opencode`	OpenCode agent
`oracle`	Reference agent (known solution)
`nop`	No-op (for testing)

Harbor Job batch scan script

#!/usr/bin/env python3
# 用法：python3 scan_job.py ./jobs/<job_name>
import json, sys
from pathlib import Path

job_dir = Path(sys.argv[1])
trials = sorted([d for d in job_dir.iterdir() if d.is_dir()])

print(f"Job: {job_dir.name}")
print(f"{'Trial':<40} {'Reward':<8} {'Status':<12} {'Duration':<10} {'Issue'}")
print("-" * 90)

for trial_dir in trials:
    result_path = trial_dir / "result.json"
    if not result_path.exists():
        print(f"{trial_dir.name:<40} {'N/A':<8} {'NO RESULT':<12} {'-':<10}")
        continue

    try:
        r = json.loads(result_path.read_text())
    except Exception as e:
        print(f"{trial_dir.name:<40} {'ERR':<8} {'JSON ERR':<12} {'-':<10} {e}")
        continue

    reward = "N/A"
    if r.get("verifier_result") and r["verifier_result"].get("rewards"):
        rewards = r["verifier_result"]["rewards"]
        if isinstance(rewards, dict):
            reward = str(list(rewards.values())[0])
        elif isinstance(rewards, list) and rewards:
            reward = str(rewards[0])
    elif (trial_dir / "verifier" / "reward.txt").exists():
        reward = (trial_dir / "verifier" / "reward.txt").read_text().strip()

    exc = r.get("exception_info")
    status = "EXCEPTION" if exc else ("OK" if reward != "N/A" else "NO REWARD")

    duration = "-"
    if r.get("agent_execution") and r["agent_execution"].get("duration_sec"):
        duration = f"{r['agent_execution']['duration_sec']:.1f}s"

    issue = ""
    if exc:
        issue = f"{exc.get('exception_type','?')}: {str(exc.get('exception_message',''))[:50]}"

    print(f"{trial_dir.name:<40} {reward:<8} {status:<12} {duration:<10} {issue}")

About this skill

Background

Important principles

Step 1: Confirm the sandbox ID and Job type

Step 2: Locate the target Job

Step 3: Read Job information

Harbor Job

/tmp/harbor/jobs/ path (local harbor run)

/data/logs/user-defined/ path (ROCK platform)

Bash Job

Step 4: Analyze results

Harbor Job: analyze trial by trial

Bash Job: single execution analysis

Step 5: In-depth debugging (when problems are found)

Harbor Job debugging

Read detailed logs

Check Docker Container status

View Agent execution trace (as needed)

Bash Job debugging

Step 6: Generate report (must output and save to a local file)

Harbor Job report format

Bash Job report format

Common failure quick reference

Harbor Job

Bash Job

Agent type reference

Harbor Job batch scan script