Run a SWE-bench instance end-to-end: ensure the instance's Docker image is available (pull from Docker Hub if missing), start a container, run an OpenSquilla agent against the issue, collect the patch, and optionally run the official evaluation.
Prerequisites — guide the user, don't dead-end
SWE-bench mode runs the official evaluation images, so it needs the Docker
CLI. Docker is NOT a hard gate on this skill: if it is missing, the
opensquilla swebench command prints the exact install command for the
user's OS and exits. When that happens, relay the install guidance to the
user ("SWE-bench needs Docker — install it with ..., then I can run this")
instead of saying the task is impossible. Also mention that solving a
real-repository coding task (not a benchmark instance) does NOT need
Docker — that is what the code-task skill is for.
Commands
Solve one instance (auto-pulls the image when missing):
opensquilla swebench solve <instance_id> --dataset verified --json
--datasetacceptsverified,multilingual, or a full HuggingFace dataset name.- Add
--evaluateto run the official harness afterwards and report whether the patch actually resolves the instance (resolvedin the JSON output). - Add
--model <model>/--thinking <level>to pin a model; leave them off to let squilla_router decide. --timeout <seconds>defaults to 1200.
Pre-fetch an image only:
opensquilla swebench pull <instance_id>
Evaluate an existing predictions file:
opensquilla swebench eval <predictions.jsonl> --dataset verified
Reading the result
solve --json prints one JSON object: state (patch_collected is
success), patch_path, artifact_dir, resolved (true/false, or null
when --evaluate was not used), duration_seconds, usage
(cost/tokens), error.
The full artifact trail (prompt, agent log, transcript, usage, patch)
lives under artifact_dir.
What to tell the user
- Before starting: if the image is not local yet, warn that the first run pulls 1-3 GB and can take a few minutes; a solve typically takes 10-30 minutes depending on the instance and timeout.
- After finishing: report
state, whether the patch is non-empty,resolvedwhen evaluation ran, the cost fromusage, and thepatch_pathso the user can inspect the diff. - On
failed/timeout: quote theerrorfield and point at<artifact_dir>/agent_stdout.logfor the full trace.
Constraints
- Images are x86_64; on ARM hosts only locally pre-built images work.
- The run happens on this machine (the gateway host) — docker must be reachable from here, and disk usage grows with each pulled image.
- Long runs block the command; for batches beyond a handful of instances, suggest the user run the CLI directly in tmux instead of going through chat.