rock-debug

ROCK Sandbox Troubleshooting Guide

Core Principles

Read-only diagnosis: All commands in this guide are for observation only,never modify files or processes inside the sandbox. All investigation is done through rockcli (logs, history, status, read-only exec). The only exception: when the user explicitly requests fixing content inside the sandbox, user confirmation must be obtained first.

Companion skill: This guide focuses on the troubleshooting process; for specific rockcli command references, see rock-cli。

Progressive confirmation: During diagnosis, if there are guesses or inferences, they must be confirmed with the user promptly. If the user does not agree, continue analysis; only if the user agrees and consents to further analysis should you continue deeper—don't just plow ahead。

Step 1: Confirm sandbox status

The first step of all troubleshooting is to confirm the sandbox's current status:

rockcli sandbox <id> status

Status transitions:pending → alive → stopping → stopped

Enter the corresponding troubleshooting branch based on the status:

Status	Meaning	Next step
alive	Sandbox running	→ Step 2A: Liveness status troubleshooting
pending	Starting	Wait, or check whether it is stuck (if it has not become alive after more than 2 minutes, consider it a startup failure)
stopped	Stopped	→ Step 2B: Stopped status troubleshooting
stopping	Stopping	Wait for stopping to complete, then handle as stopped
Unable to get status	Incorrect ID or sandbox has been cleaned up	Confirm whether the ID is correct; the sandbox may have been automatically cleaned up (auto-clear)

---

Step 2A: Sandbox alive — Interactive troubleshooting

When the sandbox is alive, all troubleshooting methods can be used. Choose based on the problem type:

Log source reminder: In addition to sandbox internal logs (rockcli sandbox <id> log search),cluster-level logsare also an important source—they record events at the cluster level for the sandbox (request routing, scheduling, sandbox creation, sandbox stop calls, etc.):

rockcli log search -k <sandbox_id> -k <keyword> --raw -m <minutes>

Command execution failure / exec error

1. Re-execute and observe the output：

rockcli sandbox <id> exec '<原始命令>'

1. Check the environment and dependencies：

rockcli sandbox <id> exec 'which python && python --version'
rockcli sandbox <id> exec 'pip list 2>&1 | head -20'
rockcli sandbox <id> exec 'df -h && free -h'            # 磁盘/内存
rockcli sandbox <id> exec 'cat /proc/cpuinfo | head -5'  # CPU

1. Search for errors in command output logs：

rockcli sandbox <id> log search --log-file command.log -k "error" -C 5
rockcli sandbox <id> log search --log-file command.log -k "Traceback" -C 10

Service/process abnormality

1. Check process status：

rockcli sandbox <id> exec 'ps aux'
rockcli sandbox <id> exec 'netstat -tlnp 2>/dev/null || ss -tlnp'

1. View system component logs：

rockcli sandbox <id> log search --log-file rocklet.log -k "error" -C 3
rockcli sandbox <id> log search --log-file rocklet_uvicorn.log -k "error"

1. Real-time tracking(suitable for use when reproducing the problem):

rockcli sandbox <id> log tail -k "error"

Resource issues (OOM / disk full)

rockcli sandbox <id> exec 'free -h'
rockcli sandbox <id> exec 'df -h'
rockcli sandbox <id> log search -m 60 -k "OOM"
rockcli sandbox <id> log search -m 60 -k "No space left"

HTTP / network issues

rockcli sandbox <id> log search --log-file access.log -k "500"
rockcli sandbox <id> log search --log-file access.log -f "status>=400"
rockcli sandbox <id> exec 'curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/health'

Query history and replay

An alive sandbox can also have its operation history and replay viewed, to trace the process of how the problem occurred:

rockcli sandbox <id> history                    # 查看完整操作历史
rockcli sandbox <id> replay                     # 回放请求序列到新沙箱

Applicable scenarios:

• Trace back the operation sequence to locate which step caused the current abnormality
• Replay to a new sandbox for comparison to rule out environment contamination factors
• Export the operation sequence for others to reproduce

---

Step 2B: Sandbox stopped — can only check history

After the sandbox stops, exec or tail is unavailable; you can only trace back through history records and logs:

Log source reminder: In addition to the sandbox internal logs (rockcli sandbox <id> log search),cluster-level logsare also an important source—they record sandbox events at the cluster level (request routing, scheduling, creating sandboxes, stopping sandboxes, calls, etc.):

rockcli log search -k <sandbox_id> -k <keyword> --raw -m <minutes>

When sandbox internal logs are missing or incomplete, prioritize relying on cluster-level logs for tracing.

View operation history

Trace all API operations throughout the sandbox lifecycle (complete requests and responses for start/exec/upload, etc.):

rockcli sandbox <id> history

Key focus areas:

• The commands and return results of the last few exec calls
• Whether there are abnormal HTTP status codes (4xx/5xx)
• The operation timeline, to locate the point where the problem occurred

Search historical logs

Logs can still be searched after the sandbox stops:

rockcli sandbox <id> log search -k "error" -m 120      # 搜索最近 2 小时
rockcli sandbox <id> log search --log-file command.log -k "kill\|OOM\|signal"
rockcli sandbox <id> log search -f "level=ERROR"

Replay request sequence

Replay historical requests in a new sandbox to reproduce the problem:

rockcli sandbox <id> replay

Replay automatically pulls the request sequence from that sandbox's history and executes it in order. Suitable for:

• Verifying whether the problem is reproducible
• Comparative troubleshooting in a new environment
• Regression testing after confirming the fix

Download log files

Download logs locally for deeper analysis:

rockcli sandbox <id> log ls                          # 列出可用日志
rockcli sandbox <id> log download command.log        # 下载命令日志
rockcli sandbox <id> log download rocklet.log        # 下载系统日志

---

Special scenario: root cause analysis of API call failures

Symptom: The user reports that get_status / execute / create_session / run_in_session / upload / download and other APIs return errors, often accompanied by the sandbox having stopped or being unreachable.

Step 1: Search cluster-level logs for stop events

rockcli log search -k <sandbox_id> -k 'stop' --raw -m <minutes>

If the time window is too small and no logs are found, try expanding the -m parameter.

Step 2: Determine the stop reason

Stop reason	Basis for judgment
User called the stop API	Appears in the previous step's logs POST /apis/envs/sandbox/v1/stop
Automatic expiration	Exists in the logs sandbox_id:[{sandbox_id}] is expired, and the time of that entry is the stop time
Unexpected crash	OOM / crash appears in rocklet.log or command.log before the stop time

Step 3: Compare timestamps

• Stop time < user failure time → The user's own behavior is abnormal (calling stop first, then trying to use the sandbox)
• There is no user stop operation, and the sandbox died on its own → Proceed to the next step of root cause analysis

Step 4: Deep dive into root causes across multiple logs

# 系统组件日志
rockcli sandbox <id> log search --log-file rocklet.log -k "error\|OOM\|kill\|crash\|exit" -m <minutes>

# 命令执行日志
rockcli sandbox <id> log search --log-file command.log -k "error\|fail\|Exception" -m <minutes>

# HTTP 错误
rockcli sandbox <id> log search --log-file access.log -k "500\|502\|503\|504" -m <minutes>

# 完整请求时间线 + 当前状态
rockcli sandbox <id> history
rockcli sandbox <id> status

Extra: Infer suspicious commands

In addition to the above logs, also combine history with command.log，to infer which of the most recent commands executed before the sandbox stopped are suspicious commands—these commands may directly cause rocklet process corruption, OOM, or file system abnormalities.

# 拉取停止时间前 N 分钟的命令历史
rockcli sandbox <id> history -s <stop_time-N> -e <stop_time>

# 按时间段过滤命令日志
rockcli sandbox <id> log search --log-file command.log -s <stop_time-N> -e <stop_time>

Common types of suspicious commands:

• Large memory allocation: Loading an oversized model, reading a large file entirely into memory, one-time cat huge logs → triggers the OOM Killer to kill rocklet
• Fork bomb / process storm: Recursively starting child processes, parallel tests without concurrency limits → exhausting PIDs or memory
• Resource-intensive IO：dd if=/dev/zero ..., writing to temporary files in a loop, not cleaning up artifacts → after the disk is full, rocklet fails to write logs
• Direct signal operations：kill -9 / pkill accidentally harming rocklet itself or its dependent processes
• Modifying system configuration: change /etc, uninstall key libraries, adjust ulimit before triggering the scenario

Following the "progressive confirmation" principle, after inferring a suspicious command,first confirm the suspected target with the user, then decide whether to dig deeper into the specific command logic.

Step 5: When there is too much output, narrow the time window

rockcli sandbox <id> log search --log-file command.log -m 30  # 仅查最近 30 分钟
rockcli sandbox <id> log search --log-file rocklet.log -m 15

---

Special scenario: diagnosing excessive disk usage

Symptom: The sandbox disk rapidly expands within a certain time period, and the user asks for the reason.

Step 1: Obtain the command execution history or analyze command.log

# 搜索指定时间段内的命令执行历史
rockcli sandbox <id> history -s <start_time> -e <end_time>

# 搜索指定时间内的命令执行日志
rockcli sandbox <id> log search --log-file command.log -k <keyword> -s <start_time> -e <end_time>

Focus on commands executed near and before the time period when disk usage surged .

Step 2: Identify suspicious commands

Based on the command situation in Step 1, infer which commands could cause a surge in disk usage;only infer, do not dig in directly. Prioritize the following two common scenarios:

1. Unit test execution

   • Characteristics: The task is executing unit tests (for example, tests involving databases such as QuestDB), and the test logic contains a large number of disk write operations.
   • Action: If you suspect this type of cause,do not directly dig into the analysis; first list the names of the test classes and confirm with the user: "Should this test logic be analyzed to obtain the root cause?"
   • Note: Only after the user approves should you dig deeper into the specific test files.

2. Manual large data writes

   • Characteristics: The user executed commands such as dd to perform large-scale data writes.

Handling logic：

• Prioritize analysis according to the above two directions.
• If the above situations are ruled out,do not expand the analysis directions on your own, but instead confirm with the user: "Having ruled out the above two situations, should the analysis be conducted from another angle?" Only after the user approves should you analyze other potential causes based on specific command characteristics.

---

Common pitfalls

• Too much output: Narrow the time window (-m、-s、-e), add more specific keywords, or use --log-file to specify a particular log file.
• history file too large: Use grep to filter the history records for a specific API endpoint.
• Sandbox already stopped：history and log commands are still available, with no need to rely on exec.

---

Common issues quick reference

Issue	Troubleshooting command
Sandbox startup stuck in pending	status confirm status; check image/resource quota; check rocklet.log
exec command has no output/times out	log search --log-file command.log；exec 'ps aux' check whether there are zombie processes
OOM / killed	exec 'dmesg | tail'；log search -k "OOM|kill|signal"
Disk full	exec 'df -h'；exec 'du -sh /* 2>/dev/null | sort -rh | head'
HTTP 5xx	log search --log-file access.log -f "status>=500"; check whether the service process is alive
Sandbox suddenly stopped	history check the last operation;log search -f "level=ERROR"; cluster-level rockcli log search -k <id> -k 'stop' --raw -m <min>
Interface call failure to locate the reason for stopping	Cluster-level rockcli log search -k <id> -k 'stop' --raw -m <min>, compare the stop time with the failure time
Disk usage suddenly spikes	history -s <ts> -e <ts> Pull commands within the time period; prioritize suspicion of unit tests / dd large data writes
Issue cannot be reproduced	history Trace back the operation sequence;replay replay in a new sandbox

Supported log files

Log file	Description
command.log	Command output
access.log	HTTP access
rocklet.log	System components
rocklet_uvicorn.log	Uvicorn service

For detailed log command parameters, see references/log.md。