rock-agent-debug

Rock Agent Debug Skill

Via rockcli query and troubleshoot Agent Jobs running in the ROCK sandbox (Harbor Job / Bash Job), and ultimately produce a detailed report saved to disk.

Background

The ROCK sandbox supports two Job types:

Type	Config type	Trial type	Identifying features
Harbor Job	`HarborJobConfig`	`HarborTrial`	Has `orchestrator`、`datasets`、`agents`、`jobs_dir`、`experiment_id` fields
Bash Job	`BashJobConfig`	`BashTrial`	Has `script` or `script_path` fields, no harbor-related fields

Job type is automatically detected via YAML configuration: first try HarborJobConfig parsing, and if that fails, fall back to BashJobConfig。

Two troubleshooting paths

Depending on the source of information, troubleshooting is divided into two paths:

Path	Applicable scenario	Core commands
Remote query (recommended)	The user provides an experiment ID + Job ID (or only the job name); the sandbox has stopped and cannot be exec'd	`rc agent view` + `rc agent fs`
In-sandbox query	The user provides a sandbox ID and the sandbox is alive; more flexible troubleshooting is needed	`rc sandbox <id> exec`

Prefer remote query：agent view and agent fs do not depend on the sandbox being alive and can directly view job/trial metadata and files, making them more efficient.

Remote query quick flow

# 1. 查看 job 状态和 trial 列表
rockcli agent view -e <exp_id> -j <job_name>

# 2. 列出 trial 文件
rockcli agent fs ls -e <exp_id> -j <job_name> -t <task_name>

# 3. 读取关键文件
rockcli agent fs cat result.json -e <exp_id> -j <job_name> -t <task_name>
rockcli agent fs cat agent/log.txt -e <exp_id> -j <job_name> -t <task_name>

# 4. 查看执行轨迹
rockcli agent view -e <exp_id> -j <job_name> --trajectory

# 5. 查看 artifacts
rockcli agent fs artifacts -e <exp_id> -j <job_name> -t <task_name>

Important principles

Stop querying once you have enough information. Do not get stuck in a loop of "if the file isn't in the expected location, search everywhere." Criteria:

Harbor Job（/data/logs/user-defined/）：.out The file contains the complete run log, and seeing the reward is enough
Harbor Job（/tmp/harbor/jobs/）：result.json + trial.log + exception.txt is enough
Bash Job：.out logs + script exit code + artifact (if any) is enough
If the key files can explain the situation,no need to keep looking, just use the existing information to produce the report

Step 1: Confirm the troubleshooting entry point and Job type

Choose the troubleshooting path based on the information the user provides:

Case A: The user provides an experiment ID + Job ID (remote query, recommended entry point)

The most common entry point is the 实验 ID + Job ID combination.agent view and agent fs both require -e <experiment_id> to locate the job; the two parameters work together to drill down level by level:

# 0. （可选）只有实验 ID、不知道 job 名时，先列出实验下的 jobs
rockcli agent view -e <experiment_id>

# 1. 查看 job 详情（实验 ID + Job ID）
rockcli agent view -e <experiment_id> -j <job_name>

# 2. 列出 job 文件（从文件结构判断类型）
rockcli agent fs ls -e <experiment_id> -j <job_name>

When there is only a job name and no experiment ID, first use rockcli agent view -E to list all experiments, find the corresponding experiment_id and then continue.

Case B: The user provides a sandbox ID (in-sandbox query)

# 查看沙箱存活状态
rockcli sandbox <id> status

# 同时检查所有可能的路径
rockcli sandbox <id> exec 'echo "=== /tmp/harbor/jobs/ ==="; ls -lt /tmp/harbor/jobs/ 2>/dev/null | head -10; echo "=== /data/logs/user-defined/ ==="; ls -lt /data/logs/user-defined/ 2>/dev/null | head -10; echo "=== /data/logs/ ==="; ls -lt /data/logs/*.out /data/logs/*.log 2>/dev/null | head -10'

Determine the Job type:

Read the YAML configuration (preferred):rockcli sandbox <id> exec 'cat /data/logs/user-defined/<job_name>.yaml'
If the configuration contains orchestrator、datasets、agents → Harbor Job
If the configuration contains script or script_path → Bash Job
If there is no YAML, only the files under /tmp/harbor/jobs/ → Harbor Job(local harbor run)

If the user gives neither a sandbox ID nor an experiment ID, ask directly.

Step 2: Locate the target Job

When the user does not specify a job, take the latest one with content (skip empty directories).

# Harbor 路径（本地 harbor run）：找最新非空的 job
rockcli sandbox <id> exec 'for d in $(ls -t /tmp/harbor/jobs/); do if [ "$(ls /tmp/harbor/jobs/$d 2>/dev/null)" ]; then echo $d; break; fi; done'

# ROCK 平台路径：列出 .yaml 配置和 .out 日志
rockcli sandbox <id> exec 'ls -lt /data/logs/user-defined/*.yaml /data/logs/user-defined/*.out 2>/dev/null | head -10'

Step 3: Read Job information

Remote query method (agent view / agent fs)

Applicable when there is a 实验 ID + Job ID scenario,Does not depend on sandbox survival status：

# 查看 job 整体状态（含 tasks 列表、进度）
rockcli agent view -e <exp_id> -j <job_name>

# 列出 job 目录结构
rockcli agent fs ls -e <exp_id> -j <job_name>

# 读取 result.json
rockcli agent fs cat result.json -e <exp_id> -j <job_name> -t <task_name>

# 读取日志文件
rockcli agent fs cat agent/log.txt -e <exp_id> -j <job_name> -t <task_name>

# 查看完整执行轨迹
rockcli agent view -e <exp_id> -j <job_name> --trajectory

# 查看 trial artifacts
rockcli agent fs artifacts -e <exp_id> -j <job_name> -t <task_name>

Query methods inside the sandbox

Harbor Job

/tmp/harbor/jobs/ path (local harbor run)

rockcli sandbox <id> exec 'cat /tmp/harbor/jobs/<job_name>/config.json'
rockcli sandbox <id> exec 'cat /tmp/harbor/jobs/<job_name>/result.json 2>/dev/null || echo "no result yet"'
rockcli sandbox <id> exec 'cat /tmp/harbor/jobs/<job_name>/job.log 2>/dev/null'

/data/logs/user-defined/ path (ROCK platform)

# 配置（agent、model、dataset）
rockcli sandbox <id> exec 'cat /data/logs/user-defined/<job_name>.yaml'

# 完整运行日志（含进度、reward、时间线）
rockcli sandbox <id> exec 'cat /data/logs/user-defined/<job_name>.out'

# result.json（可能不存在，不要为此继续搜索）
rockcli sandbox <id> exec 'cat /data/logs/user-defined/jobs/<job_name>/result.json 2>/dev/null || echo "no result.json"'

.out The file is enough to determine whether the task is complete(including the reward line, completion time, trial name), do not, just because result.json does not exist, search everything.

Bash Job

# 配置（script / script_path）
rockcli sandbox <id> exec 'cat /data/logs/user-defined/<job_name>.yaml'

# 完整运行日志
rockcli sandbox <id> exec 'cat /data/logs/user-defined/<job_name>.out'

# 检查是否有 artifact 输出
rockcli sandbox <id> exec 'ls -lt /data/logs/artifacts/ 2>/dev/null | head -10'

Characteristics of Bash Job:

Single script execution, no concept of trial
The exit code is the key to determining success/failure
There may be OSS artifact output to artifacts/{namespace}/{experiment_id}/{job_name}/
Logs are output directly to the .out file

Step 4: Analyze results

Harbor Job: Trial-by-trial analysis

Trial file structure (/tmp/harbor/jobs/ path):

<trial_name>/
├── result.json          # 完整结果（阶段时间、reward、异常）
├── trial.log            # 简短摘要（失败时有错误原因）
├── exception.txt        # 完整异常栈
├── agent/setup/stdout.txt   # Agent 安装日志
├── agent/setup/return-code.txt
├── verifier/reward.txt
├── verifier/test-stdout.txt
├── verifier/test-stderr.txt
└── agent/trajectory.json    # Agent 执行轨迹（ATIF 格式，若有）

# 列出 trials
rockcli sandbox <id> exec 'ls /tmp/harbor/jobs/<job_name>/ 2>/dev/null || ls /data/logs/user-defined/jobs/<job_name>/ 2>/dev/null'

# 批量查看所有 trial 状态（快速）
rockcli sandbox <id> exec 'for t in /tmp/harbor/jobs/<job_name>/*/; do echo "=== $(basename $t) ==="; cat "$t/result.json" 2>/dev/null | python3 -c "import json,sys; d=json.load(sys.stdin); print(\"reward:\", d.get(\"verifier_result\"), \"exception:\", bool(d.get(\"exception_info\")))" 2>/dev/null || cat "$t/trial.log" 2>/dev/null | head -3; done'

Explanation of key fields in result.json:

Field	Meaning	Abnormal signal
`exception_info`	Exception type and message	Non-null indicates an uncaught exception
`verifier_result.rewards`	Individual scores	All 0 or null indicates the verifier did not run
`agent_result`	Agent execution result	null indicates the agent did not complete normally
`agent_execution.duration_sec`	Agent runtime duration	Extremely short (<5s) indicates an early exit
`environment_setup`	Environment startup duration	Extremely long indicates an image build or startup problem
`started_at` / `finished_at`	Start/end time	finished_at being null indicates it is not complete

Determine the current stage of the Trial (from result.json):

environment_setup has a value,agent_setup is null → environment is starting
agent_setup has a value,agent_execution is null → Agent is installing
agent_execution has a value,verifier is null → Agent is executing
verifier has a value,finished_at is null → Verifier is running
finished_at has a value → completed

Bash Job: Single execution analysis

Bash Job has no concept of trial; it is a one-off script execution:

# 查看退出码（.out 末尾通常有 exit code）
rockcli sandbox <id> exec 'tail -20 /data/logs/user-defined/<job_name>.out'

# 搜索错误
rockcli sandbox <id> exec 'grep -i "error\|exception\|failed\|traceback" /data/logs/user-defined/<job_name>.out | tail -30'

# 如果有 artifact，检查输出
rockcli sandbox <id> exec 'ls -R /data/logs/artifacts/<namespace>/<experiment_id>/<job_name>/ 2>/dev/null'

Step 5: In-depth debugging (when problems are found)

Harbor Job debugging

Read detailed logs

# 1. 如果有 exception.txt，先读它
rockcli sandbox <id> exec 'cat /tmp/harbor/jobs/<job_name>/<trial_name>/exception.txt'

# 2. Verifier 输出（测试脚本的实际运行结果）
rockcli sandbox <id> exec 'cat /tmp/harbor/jobs/<job_name>/<trial_name>/verifier/test-stdout.txt'
rockcli sandbox <id> exec 'cat /tmp/harbor/jobs/<job_name>/<trial_name>/verifier/test-stderr.txt'

# 3. Trial 主日志（先搜关键词）
rockcli sandbox <id> exec 'grep -i "error\|exception\|failed\|traceback" /tmp/harbor/jobs/<job_name>/<trial_name>/trial.log | tail -50'

Check Docker Container status

Harbor Docker container naming rule:hb__<environment_name>(lowercase)

# 列出所有 Harbor 相关 container
rockcli sandbox <id> exec 'docker ps -a | grep "hb__"'

# 查看 container 详情
rockcli sandbox <id> exec 'docker inspect <container_name> | python3 -c "
import json, sys
data = json.load(sys.stdin)[0]
state = data[\"State\"]
print(f\"Status: {state[chr(39)]Status{chr(39)}}\")
print(f\"ExitCode: {state[chr(39)]ExitCode{chr(39)}}\")
print(f\"OOMKilled: {state[chr(39)]OOMKilled{chr(39)}}\")
"'

# 查看 container 最近日志
rockcli sandbox <id> exec 'docker logs <container_name> --tail 100'

View the Agent execution trajectory (as needed)

Only check when the user wants to know what the agent did.

rockcli sandbox <id> exec 'cat /path/to/trial/agent/trajectory.json | python3 -c "
import json, sys
d = json.load(sys.stdin)
steps = d.get(\"steps\", [])
print(f\"共 {len(steps)} 步\")
for i, s in enumerate(steps[:20]):
    tool = s.get(\"tool\", \"\") or s.get(\"action\", \"\")
    summary = s.get(\"observation\", \"\") or s.get(\"result\", \"\")
    print(f\"  [{i+1}] {tool}: {str(summary)[:100]}\")
"'

Bash Job debugging

# 1. 查看完整日志
rockcli sandbox <id> exec 'cat /data/logs/user-defined/<job_name>.out'

# 2. 检查沙箱内执行环境
rockcli sandbox <id> exec 'echo "=== Python ==="; python3 --version 2>/dev/null; echo "=== Node ==="; node --version 2>/dev/null; echo "=== Disk ==="; df -h / 2>/dev/null'

# 3. 如果有 artifact，查看内容
rockcli sandbox <id> exec 'cat /data/logs/artifacts/<namespace>/<experiment_id>/<job_name>/* 2>/dev/null | head -100'

Step 6: Generate a report (must output and save to a local file)

At the end of every analysis, you must:

Output a structured report in the conversation
Save the report to a local file:/tmp/rock-agent-report-<sandbox_id_短>-<timestamp>.md

Harbor Job report format

## Harbor Job 报告

**沙箱**: <sandbox_id>
**Job**: <job_name>
**类型**: Harbor Job
**时间**: <started_at> → <finished_at 或 "运行中">
**耗时**: <duration>

### 配置
- Agent: <agent_name> (<model_name>)
- Bench: <dataset_name> v<version>（或 Task: <task_path>）
- 并发数: <n_concurrent_trials>
- Verifier 模式: <harbor/native>

### 整体结果
- 总 Trials: <n>，成功: <n>，失败: <n>
- 平均 Reward: <mean>

### Trial 明细
| Trial | 状态 | Reward | 耗时 | 问题 |
|-------|------|--------|------|------|
| <name> | ✅/❌/🔄/⚠️ | <reward> | <duration> | <issue 或 -> |

### 执行时间线
- `<HH:MM>` 环境启动完成（耗时 Xs）
- `<HH:MM>` Agent 安装完成（耗时 Xs）
- `<HH:MM>` Agent 开始执行
- `<HH:MM>` Verifier 完成，reward=<N>

### 问题分析（如有）

**[CRITICAL] 问题标题**
- 影响 trial：`<trial_name>`
- 根本原因：...
- 证据：`exception.txt` 关键日志
- 建议：...

### 行动建议
1. **立即修复**：...
2. **下次运行前**：...

Bash Job report format

## Bash Job 报告

**沙箱**: <sandbox_id>
**Job**: <job_name>
**类型**: Bash Job
**时间**: <started_at> → <finished_at>
**耗时**: <duration>
**退出码**: <exit_code>

### 配置
- Script: <script 内容或 script_path>
- Environment: <environment 配置>
- Timeout: <timeout>

### 执行结果
- 状态: ✅ 成功 / ❌ 失败
- 退出码: <0 或非零>
- 关键输出: <摘要>

### 日志摘要
<关键日志片段，特别是错误信息>

### 问题分析（如有）
- 根本原因：...
- 证据：日志关键行
- 建议：...

### 行动建议
1. ...

Status icons: ✅ Passed | ❌ Failed | 🔄 Running | ⚠️ Completed but abnormal

Common fault quick reference

Harbor Job

Symptom	Failure stage	Typical cause
"Docker compose command failed" / "no such host"	env_setup	Image pull DNS failure
`agent/setup/return-code.txt` = 100	agent_setup	Agent installation script failure
`AgentTimeoutError`	agent_execution	Agent timeout
reward=0, no exception	verifier	Agent completed but did not solve the problem
`RewardFileNotFoundError`	verifier	reward.txt does not exist
`no running event loop`	cleanup	Cleanup phase crashes after Agent timeout (not the primary failure cause)
reward.txt is empty or does not exist	verifier	verifier did not run to the reward-writing step
agent_result is null, execution time is extremely short	agent_setup	agent installation script error
All trials failed, same exception	Environment level	Image/dependency/network issues

Bash Job

Symptom	Typical cause
Non-zero exit code	Script execution failed
Exited due to timeout	`timeout` setting too short or script hung
Log truncated	Insufficient sandbox resources (OOM / disk full)
No artifact output	Script did not write to the expected path or path permission issue

Agent Type Reference

Agent name	Description
`claude-code`	Claude Code CLI
`terminus` / `terminus-2`	Terminus internal agent
`swe-agent-internal`	Internal SWE-Agent implementation
`openhands`	OpenHands agent
`mini-swe-agent`	Lightweight SWE-Agent
`aider`	Aider coding agent
`opencode`	OpenCode agent
`oracle`	Reference agent (known solution)
`nop`	No-op (for testing)

Harbor Job batch scanning script

#!/usr/bin/env python3
# 用法：python3 scan_job.py ./jobs/<job_name>
import json, sys
from pathlib import Path

job_dir = Path(sys.argv[1])
trials = sorted([d for d in job_dir.iterdir() if d.is_dir()])

print(f"Job: {job_dir.name}")
print(f"{'Trial':<40} {'Reward':<8} {'Status':<12} {'Duration':<10} {'Issue'}")
print("-" * 90)

for trial_dir in trials:
    result_path = trial_dir / "result.json"
    if not result_path.exists():
        print(f"{trial_dir.name:<40} {'N/A':<8} {'NO RESULT':<12} {'-':<10}")
        continue

    try:
        r = json.loads(result_path.read_text())
    except Exception as e:
        print(f"{trial_dir.name:<40} {'ERR':<8} {'JSON ERR':<12} {'-':<10} {e}")
        continue

    reward = "N/A"
    if r.get("verifier_result") and r["verifier_result"].get("rewards"):
        rewards = r["verifier_result"]["rewards"]
        if isinstance(rewards, dict):
            reward = str(list(rewards.values())[0])
        elif isinstance(rewards, list) and rewards:
            reward = str(rewards[0])
    elif (trial_dir / "verifier" / "reward.txt").exists():
        reward = (trial_dir / "verifier" / "reward.txt").read_text().strip()

    exc = r.get("exception_info")
    status = "EXCEPTION" if exc else ("OK" if reward != "N/A" else "NO REWARD")

    duration = "-"
    if r.get("agent_execution") and r["agent_execution"].get("duration_sec"):
        duration = f"{r['agent_execution']['duration_sec']:.1f}s"

    issue = ""
    if exc:
        issue = f"{exc.get('exception_type','?')}: {str(exc.get('exception_message',''))[:50]}"

    print(f"{trial_dir.name:<40} {reward:<8} {status:<12} {duration:<10} {issue}")

About this skill

Background

Two troubleshooting paths

Remote query quick flow

Important principles

Step 1: Confirm the troubleshooting entry point and Job type

Case A: The user provides an experiment ID + Job ID (remote query, recommended entry point)

Case B: The user provides a sandbox ID (in-sandbox query)

Step 2: Locate the target Job

Step 3: Read Job information

Remote query method (agent view / agent fs)

Query methods inside the sandbox

Harbor Job

/tmp/harbor/jobs/ path (local harbor run)

/data/logs/user-defined/ path (ROCK platform)

Bash Job

Step 4: Analyze results

Harbor Job: Trial-by-trial analysis

Bash Job: Single execution analysis

Step 5: In-depth debugging (when problems are found)

Harbor Job debugging

Read detailed logs

Check Docker Container status

View the Agent execution trajectory (as needed)

Bash Job debugging

Step 6: Generate a report (must output and save to a local file)

Harbor Job report format

Bash Job report format

Common fault quick reference

Harbor Job

Bash Job

Agent Type Reference

Harbor Job batch scanning script