rock-agent-sdk

ROCK Agent SDK Development Guide

Help developers quickly write Agent configurations, create Benchmark evaluation Jobs, and run and analyze results based on the ROCK Agent SDK (rock.sdk.agent).

ROCK Platform Background

ROCK (Reinforcement Open Construction Kit) is a sandbox environment management framework developed by Alibaba, designed specifically for Agentic reinforcement learning scenarios. It adopts a client-server architecture:

Component	Responsibility
ROCK SDK	Environment development toolkit, providing APIs for Agent development, Job configuration, Benchmark evaluation, etc.
ROCK CLI (rockcli)	Command-line tool for managing sandboxes, executing commands, and uploading/downloading files
ROCK Admin	Scheduling node, responsible for sandbox deployment and resource scheduling
ROCK Worker	Worker node, allocating physical resources to sandboxes and executing the runtime
ROCK Envhub	Environment repository, providing registration and storage of environment data

Harbor is an Agent Benchmark evaluation framework that runs in ROCK sandboxes. Developers write JobConfigthrough the ROCK Agent SDK, and ROCK creates a sandbox and runs the Harbor evaluation task within it, ultimately collecting Trial results (reward, trajectory, logs, etc.).

SDK Core Concepts

Package Path

rock.sdk.agent/
├── __init__.py              # 导出 Job, JobConfig, JobResult 等
├── constants.py             # 常量（超时、日志路径）
├── job.py                   # Job 类（提交/运行/取消评测任务）
└── models/
    ├── environment_type.py  # 环境类型枚举
    ├── orchestrator_type.py # 编排器类型枚举
    ├── job/
    │   ├── config.py        # JobConfig 及所有子配置
    │   └── result.py        # JobResult, JobStatus
    ├── metric/
    │   ├── config.py        # MetricConfig
    │   └── type.py          # MetricType 枚举
    └── trial/
        ├── config.py        # AgentConfig, VerifierConfig, TaskConfig 等
        └── result.py        # TrialResult, AgentInfo, VerifierResult

Core Execution Flow

Job(config).run()
  → 创建 ROCK 沙箱
  → 上传配置和脚本到沙箱
  → 生成 Harbor YAML + 运行脚本
  → nohup 异步执行 harbor jobs start
  → 轮询进程状态（30s 间隔）
  → 收集每个 Trial 的 result.json
  → 返回 JobResult

Three Job Running Modes

# 模式 1：完整运行（提交 + 等待）
result = await Job(config).run()

# 模式 2：异步提交 + 等待
job = Job(config)
await job.submit()       # 立即返回
# ... 做其他事情 ...
result = await job.wait()

# 模式 3：取消运行中任务
await job.cancel()

Configuration Writing Guide

Method 1: YAML Configuration File

The most commonly used method, suitable for most scenarios.

Complete Configuration Template

# ============ Job 身份 ============
experiment_id: "my-experiment"           # 必填：实验标识
job_name: "swe-bench-test"              # 可选：Job 名称（不设置则自动生成）
namespace: "my-team"                    # 可选：租户隔离

# ============ ROCK 环境 ============
environment:
  base_url: "<your-rock-base-url>"      # ROCK 服务地址
  # 认证（二选一）
  xrl_authorization: "<token>"          # 已弃用，推荐用 extra_headers
  extra_headers:
    Authorization: "Bearer <token>"
  # 沙箱配置
  image: "python:3.11"                  # Docker 镜像（含 harbor 时使用 harbor 镜像）
  cluster: "zb"                         # 集群标识
  memory: "16g"                         # 内存限制
  cpus: 4                               # CPU 限制
  startup_timeout: 600                  # 启动超时（秒）
  auto_clear_seconds: 3600              # 自动清理时间
  auto_stop: true                       # 完成后自动停止沙箱
  # 沙箱启动后执行的命令
  setup_commands:
    - "echo 'setup done'"
  # 运行前上传的文件
  file_uploads: []
  # 环境变量（API Key 等）
  env:
    OPENAI_API_KEY: "<your-key>"
    OPENAI_BASE_URL: "<your-base-url>"

# ============ Agent 配置 ============
agents:
  - name: "claude-code"                 # Agent 名称
    model_name: "custom_openai/gpt-4o"  # 模型名称
    override_timeout_sec: 600           # 单次超时
    max_timeout_sec: 1800               # 最大超时
    kwargs: {}                          # 额外参数
    env: {}                             # Agent 专属环境变量

# ============ 编排器 ============
orchestrator:
  type: "local"                         # local（本地）或 queue（队列）
  n_concurrent_trials: 4                # 并发 Trial 数
  quiet: false
  retry:
    max_attempts: 1                     # 失败重试次数

# ============ 验证器 ============
verifier:
  override_timeout_sec: 300
  max_timeout_sec: 600
  disable: false
  mode: "harbor"                        # harbor（默认）或 native（容器化验证器）

# ============ 数据集 ============
datasets:
  - name: "princeton-nlp/SWE-bench_Verified"
    registry:
      split: "test"
      # OSS 数据集注册中心（推荐）
      oss_access_key_id: "<key>"
      oss_access_key_secret: "<secret>"
      oss_bucket: "<bucket>"
      oss_dataset_path: "<path>"
      oss_region: "<region>"
      oss_endpoint: "<endpoint>"
    task_names: []                      # 空=全部，指定则只跑这些
    exclude_task_names: []              # 排除的任务
    n_tasks: null                       # 限制任务数量

# ============ 指标 ============
metrics: []

# ============ 标签 ============
labels:
  team: "my-team"
  version: "v1"

Common Scenario Templates

SWE-bench Verified evaluation:

experiment_id: "swe-bench-<model-name>"
environment:
  base_url: "<rock-url>"
  image: "<harbor-image:tag>"
  cluster: "zb"
  memory: "32g"
  cpus: 8
  startup_timeout: 1800
  auto_stop: true
  env:
    OPENAI_API_KEY: "<key>"
    OPENAI_BASE_URL: "<url>"

agents:
  - name: "swe-agent"
    model_name: "custom_openai/<model>"

orchestrator:
  n_concurrent_trials: 4

datasets:
  - name: "princeton-nlp/SWE-bench_Verified"
    registry:
      split: "test"
      # OSS 配置（或使用环境变量自动转发）
    task_names:
      - "astropy__astropy-7606"

Terminal Bench 2 evaluation:

experiment_id: "terminal-bench-<model-name>"
environment:
  base_url: "<rock-url>"
  image: "<harbor-image:tag>"
  cluster: "zb"
  memory: "16g"
  cpus: 4
  auto_stop: true
  env:
    LLM_API_KEY: "<key>"
    LLM_BASE_URL: "<url>"

agents:
  - name: "openhands"
    model_name: "<provider>/<model>"

orchestrator:
  n_concurrent_trials: 1

datasets:
  - name: "terminal-bench-2-test"
    registry:
      split: "test"

Method 2: Python SDK

Suitable for scenarios requiring programmatic configuration and dynamic parameters.

import asyncio
from rock.sdk.agent import Job, JobConfig
from rock.sdk.agent.models.job.config import (
    RockEnvironmentConfig,
    AgentConfig,
    VerifierConfig,
    OrchestratorConfig,
    DatasetConfig,
    RegistryDatasetConfig,
)
from rock.sdk.sandbox.config import SandboxConfig

async def main():
    # 构建配置
    config = JobConfig(
        experiment_id="my-experiment",
        environment=RockEnvironmentConfig(
            base_url="<rock-url>",
            image="python:3.11",
            cluster="zb",
            memory="16g",
            cpus=4,
            auto_stop=True,
            env={"OPENAI_API_KEY": "<key>"},
        ),
        agents=[
            AgentConfig(
                name="my-agent",
                model_name="custom_openai/gpt-4o",
            )
        ],
        orchestrator=OrchestratorConfig(n_concurrent_trials=4),
        verifier=VerifierConfig(mode="harbor"),
        datasets=[
            # 通过 Registry 加载数据集
        ],
    )

    # 运行评测
    result = await Job(config).run()

    # 分析结果
    print(f"Score: {result.score}")
    print(f"Completed: {result.n_completed}/{len(result.trial_results)}")
    for trial in result.trial_results:
        print(f"  {trial.task_name}: {trial.score} ({trial.status})")

asyncio.run(main())

Method 3: Load from YAML and Modify

from rock.sdk.agent import Job, JobConfig

# 加载 YAML 配置
config = JobConfig.from_yaml("job_config.yaml")

# 动态修改
config.datasets[0].task_names = ["specific-task-1", "specific-task-2"]
config.environment.memory = "32g"

# 序列化回 YAML（不含 Rock 环境字段，纯 Harbor 格式）
harbor_yaml = config.to_harbor_yaml()

Agent Development

rock_agent_config.yaml

Each Agent requires a rock_agent_config.yaml configuration file, defining the installation and running method:

# Agent 运行命令（${prompt} 会被替换为实际输入）
run_cmd: "claude -p ${prompt}"

# 运行时环境
runtime_env_config:
  type: node                           # node 或 python
  custom_install_cmd: "npm install -g @anthropic-ai/claude-code"
  npm_registry: "https://registry.npmmirror.com"  # 可选：npm 镜像

# 环境变量
env:
  ANTHROPIC_BASE_URL: "<url>"
  ANTHROPIC_API_KEY: "<key>"

# 可选配置
skip_wrap_run_cmd: false               # 跳过 PATH 包装
project_path: "/workspace"             # Agent 工作目录
working_dir: "/workspace"
post_init_cmds:                        # 初始化后执行的命令
  - "git config --global user.email 'bot@example.com'"

Examples of Supported Agents

Agent	Installation Command	run_cmd
Claude Code	npm install -g @anthropic-ai/claude-code	claude -p ${prompt}
Cursor CLI	curl https://cursor.com/install | bash	cursor-agent -p ${prompt}
Qwen Code	npm install -g @qwen-code/qwen-code@latest	qwen -y ${prompt}
OpenHands	Install via pip	openhands run
SWE-Agent	pip install -e .	sweagent run

Lightweight Agent Run (Non-Benchmark)

Suitable for quickly testing Agents without the full Benchmark process:

import asyncio
from rock.sdk.sandbox.client import Sandbox
from rock.sdk.sandbox.config import SandboxConfig

async def main():
    sandbox = Sandbox(SandboxConfig())
    await sandbox.start()
    await sandbox.agent.install()    # 根据 rock_agent_config.yaml 安装
    result = await sandbox.agent.run("Hello, solve this bug")
    print(f"result={result}")
    await sandbox.stop()

asyncio.run(main())

Configuration Details

Dataset Registry Types

Type	Description	Applicable Scenarios
OssRegistryInfo	Alibaba Cloud OSS Dataset	Internal Benchmark (Recommended)
RemoteRegistryInfo	Remote Registry (Default GitHub)	Open-source Benchmark
LocalRegistryInfo	Local File System	Local development and debugging

Environment Types

Type	Description
docker	Docker Container (Default)
rock	ROCK Sandbox
daytona / e2b / modal / runloop / gke	Third-party Sandbox

Orchestrator Types

Type	Description
local	Local orchestration, Trials execute sequentially/concurrently within the sandbox
queue	Queue orchestration, suitable for large-scale distributed evaluation

Result Analysis

JobResult Structure

class JobResult:
    job_id: str
    status: JobStatus           # pending/running/completed/failed/cancelled
    labels: dict[str, str]
    trial_results: list[TrialResult]
    raw_output: str
    exit_code: int

    @property
    def score(self) -> float    # 所有 Trial 的平均分
    @property
    def n_completed(self) -> int
    @property
    def n_failed(self) -> int

TrialResult Structure

class TrialResult:
    task_name: str
    trial_name: str
    agent_info: AgentInfo       # Agent 名称、版本、模型
    agent_result: AgentResult   # token 使用量、成本、rollout
    verifier_result: VerifierResult  # rewards 字典
    exception_info: ExceptionInfo    # 异常信息

    @property
    def score(self) -> float    # verifier_result.rewards["reward"]
    @property
    def status(self) -> str     # "failed" 或 "completed"
    @property
    def duration_sec(self) -> float

Result Analysis Example

result = await Job(config).run()

# 总分
print(f"平均分: {result.score}")

# 逐 Trial 分析
for trial in result.trial_results:
    if trial.status == "failed":
        print(f"❌ {trial.task_name}: {trial.exception_info.exception_type}")
    else:
        print(f"✅ {trial.task_name}: reward={trial.score}, 耗时={trial.duration_sec:.1f}s")

# 失败率
print(f"失败率: {result.n_failed}/{len(result.trial_results)}")

Automatic Forwarding of Environment Variables

The following OSS environment variables are automatically forwarded from the host process to the sandbox and do not need to be written in the YAML:

• OSS_ACCESS_KEY_ID
• OSS_ACCESS_KEY_SECRET
• OSS_REGION
• OSS_ENDPOINT
• OSS_BUCKET
• OSS_DATASET_PATH

FAQ

Q: How do I choose the memory and CPU configuration?

Benchmark	Recommended memory	Recommended cpus
SWE-bench Verified	32g	8
Terminal Bench 2	16g	4
Lightweight testing	8g	2

Q: How do I set timeout_multiplier?

The default is 1.0. If the Agent frequently times out, you can set it to 1.5~2.0. Total wait timeout = agent_timeout * multiplier + 600s (environment startup + data download + verifier buffer), with a default fallback cap of 7200s (2 hours).

Q: How do I run only specific tasks?

In datasets specify task_names：

datasets:
  - name: "princeton-nlp/SWE-bench_Verified"
    registry: { ... }
    task_names:
      - "astropy__astropy-7606"
      - "django__django-12345"

Q: How do I use Native Verifier?

verifier:
  mode: "native"

Native mode supports containerized verifiers and is suitable for testing scenarios that require an independent environment.

Q: What should I do if the OSS dataset configuration is too long?

Use automatic environment variable forwarding; just export the OSS environment variables before running, and the OSS field can be omitted in the YAML.