Rent a GPU that is ready in seconds, not minutes. The whole method is one rule: everything installable goes into a prebaked container image, so the pod's only job at boot is to pull that image, start sshd, and run the workload. Pod-side apt install, pip install, and source compilation are all billed minutes that buy nothing.
When to Use
- Work that is impossible or impractically slow on local hardware: needs more VRAM than the workstation has, needs CUDA specifically, needs N GPUs on one host.
- Benchmark grids, fine-tuning and distillation runs, short-lived model serving.
- Any repeat rental: the second pod is where a prebaked image pays for itself.
Do not rent for work that fits locally. Write the case for the pod first (what is locally impossible, what decides when it is done), and give the workload a hard stop before launching it.
The core rule: prebake, never install on the pod
| Stage | Prebaked image | Stock image plus pod-side installs |
|---|---|---|
| Boot to verified environment, warm host (image cached) | 16 s | 8-10 min |
| Boot to verified environment, cold host (1.9 GB image pull) | 64 s | 8-10 min |
Measured 2026-08-12 on RunPod secure cloud; "verified" means pod created, RUNNING, ssh accepted, and an on-pod check of GPU count, binary version, and the prebaked virtual environment all passed. Related measurements from the same session:
- A 17 GB model file pulled from HuggingFace on the pod in 34.8 s (about 490 MB/s) with
hf_transferpreinstalled and enabled. Datacenter network beats any local upload. - 2x A40 at $0.88/hr total served a 30B model layer-split across both cards (8.2 + 8.6 GiB resident) at 31 tok/s decode.
Always report the GPU type, GPU count, hourly price, and the date next to any number like these. Prices and stock move, and a throughput number without its machine spec is not a result.
Why RunPod over a managed serverless platform
Compared to managed alternatives such as Modal, RunPod runs roughly half the price for the same card, needs minimal prep work, exposes a usable REST and GraphQL API, and sits on a fast datacenter network. The trade-off is that the image, the lifecycle, and the cost discipline are yours to manage, which is exactly what the templates in this skill do.
Workflow
Step 1: Pick the GPU and confirm it is in stock
Availability is per (GPU type, GPU count): a type with stock at 1x may have none at 2x. Query before assuming, because a create call against an unavailable type fails with "There are no instances currently available".
curl -s https://api.runpod.io/graphql \
-H "Authorization: Bearer $RUNPOD_API_KEY" -H 'Content-Type: application/json' \
-d '{"query":"{ gpuTypes { id securePrice lowestPrice(input:{gpuCount:1, secureCloud:true}) { stockStatus uninterruptablePrice } } }"}'
The REST API has no gputypes path; stock and pricing come from GraphQL only. GraphQL returns HTTP 200 even for query errors, so check the body for an errors key rather than trusting the status code. For the pricing ladder, the aggregate-VRAM-per-dollar comparison, and the scale ladder from one card to a multi-node cluster, see references/gpu-selection.md.
Step 2: Build the prebaked image
Start from templates/Dockerfile. Five rules make it fast and stable:
- Match the base image glibc to any prebuilt binaries copied in. Binaries built on Ubuntu 24.04 will not run on a 22.04 base.
- Never compile, in the image or on the pod.
COPY --froma digest-pinned official prebuilt image instead. Compiling CUDA code underbuildxemulation on Apple Silicon is painfully slow; copying binaries is instant. - Prebake the dependency environment from the lockfile (
uv sync --frozen --no-install-project --no-dev), so the post-sync install on the pod is a warm-cache no-op. Never let a tool's ownrequirements.txtdrag in a second deep-learning framework install; that has filled a 60 GB disk before. - Preinstall
hf_transfer, setHF_HUB_ENABLE_HF_TRANSFER=1, and pointHF_HOMEat the container volume so weight pulls run at datacenter speed and survive into the workspace. - Publish environment for
ssh, not just for Docker.sshdspawns clean login shells that do not inherit DockerfileENV, so register library paths withldconfigand write the variables to/etc/environment.
Validate locally before pushing. This gate costs nothing on a laptop and would otherwise be paid in billed pod-debugging minutes:
docker buildx build --platform linux/amd64 -f Dockerfile -t "$IMAGE" --push .
docker run --platform linux/amd64 --rm "$IMAGE" <the-binary> --version # runtime linkage check
docker run --platform linux/amd64 --rm "$IMAGE" ls /opt/<env-dir>/.venv # prebaked env present
Step 3: Create the pod
templates/pod-up.sh creates the pod and prints a timestamped marker for every stage, exiting nonzero on the first failure. Four fields in the create call matter:
env: {"PUBLIC_KEY": "<contents of the local .pub key>"}. Pods created through the REST API get no auto-injectedsshkey; RunPod only injects account keys into its own templates.allowedCudaVersions: ["12.6","12.7","12.8","12.9"], pinning away from a host pool with a known container-start bug (see the pitfall catalog).containerDiskInGbsized for two copies of the model plus caches.gpuCount: Nfor a single-host multi-GPU pod. Same call, same pod, N cards.
Step 4: Sync the project and launch the job detached
templates/pod-run.sh has three modes: launch, status, fetch. Long jobs run detached under tmux with an explicit exit-code marker line, so the local session can disconnect without killing the run. pod-up.sh ends by printing the export POD_HOST=... POD_PORT=... line that pod-run.sh requires; export it before any mode.
Two non-obvious flags carry real scar tissue: rsync into a container fails chown (exit code 23) and kills a set -e script even though every file transferred, so always pass --no-owner --no-group. And every launch must be marker-checked, never fire-and-forget: echo an explicit STAGE / READY / FAILED marker and verify it. A launch that silently no-opped on a renamed path once burned 17 idle billed minutes.
Never suppress stderr on an orchestration path. A 2>/dev/null on a launch ssh hid a self-killing pkill for half an hour, disguised as network flakiness (pitfall 17). Capture per-pod logs instead.
Step 5: Fetch results, then terminate immediately
Copy results back to the local machine with scp or rsync. Do not stage them through a model hub; free-plan private storage caps have failed uploads mid-chain. Then templates/pod-down.sh, which reads the pod id recorded by pod-up.sh.
Terminating is part of finishing the job, not a separate cleanup task. Full detached-run, marker, and cost protocol: references/job-execution.md.
Fanning out and scaling
An evaluation grid or batch sweep is embarrassingly parallel: N independent single-GPU pods burn the same total GPU-hours as one pod and finish in 1/N of the wall-clock time. Reach for a fleet of independent pods, not a multi-node instant cluster: clusters are distributed-training infrastructure (collective communication, interconnect, launcher config) and the wrong shape for independent cells. For the rare run that genuinely needs a cluster, the scale ladder in references/gpu-selection.md includes the rung-4 pre-flight checklist: provision it as an instant cluster rather than hand-wired pods, prove the fabric with an NCCL all-reduce before the real job, make checkpoint-resume mandatory, and rehearse the full launch at the smallest shape because debugging bills every node at once.
Nine rules carry most of the value; the full method is in references/fanout.md.
- Slice by estimated duration, not item count. A long-context bin can run 10x a short one, so equal item counts leave most of the fleet idle. Chain several short runs onto one pod (
cmd1; cmd2; cmd3inside onetmuxsession) and write one results file per slice, merged locally after fetch. - Provision what stock allows, then queue. Availability is per (GPU type, GPU count), so create pods until stock runs out and round-robin the unplaced slices onto the pods that did provision, rebalanced by duration. Do not stall a grid waiting for the last pod.
- Verify one pod end to end before replicating. Session live, log line count advancing between two samples, GPU utilization engaged. Only then copy the launch to the rest of the fleet; an unverified launch multiplies one mistake by the size of the fleet.
- Launchers are detached retry-with-backoff loops with a verification connection after every attempt. A transient network window can fail every pod in one wave and look exactly like a systematic bug.
ssh -nin every loop. Without itssheats the loop's stdin and only the first iteration runs, while the rest of the fleet boots and bills doing nothing.- Build command chains in Python, not in the shell. zsh does not word-split unquoted variables the way bash does, so a chain assembled from a variable can collapse into one malformed command.
- Make relaunches idempotent. A retry loop that can fire twice must be safe to fire twice: kill the previous session and any orphaned worker first (with the character-class dodge, so the kill cannot match its own command line). An orphaned worker also holds the GPU busy, turning "GPU engaged" into a false launch-verification signal.
- Monitor from one detached loop, and calibrate an ETA from measured throughput. End remote content pipelines with
; exit 0(a zero-matchgrepotherwise reads as an unreachable pod), forcePYTHONUNBUFFERED=1so logs advance in real time, and convert an early tokens-per-second reading into per-cell ETA so silence can be told apart from a stall. Declare a pod unreachable only when a dedicated probe fails. - Reap on completion. A finished pod bills until something terminates it, and a monitor that only reports does not stop the meter. Run a local reaper loop: probe, detect chain-process-gone, fetch results and log, verify locally, and only then delete the pod. Never terminate on a failed fetch or an unreachable probe.
Per-pod startup runs about 4 min when the image is only partially prebaked (dependency sync, model pull, evaluation-dataset pulls, server load) and drops to about 90 s once the datasets and all dependency groups are baked in. That difference is paid once at build time and saved N times across the fleet.
Templates
| File | Purpose |
|---|---|
| templates/Dockerfile | Prebaked CUDA image: matched glibc base, copied prebuilt binaries, locked dependency environment, ssh-visible environment |
| templates/start.sh | Entrypoint: inject PUBLIC_KEY, start sshd, print the ready marker, idle |
| templates/pod-up.sh | Create the pod, wait for RUNNING and ssh, verify the environment, print boot-to-ready timings |
| templates/pod-run.sh | launch / status / fetch: rsync the project, run detached under tmux, copy results back |
| templates/pod-down.sh | Terminate the pod recorded by pod-up.sh |
Every template is parameterized through environment variables (RUNPOD_API_KEY, POD_IMAGE, POD_SSH_KEY, POD_NAME, POD_HOST, POD_PORT, POD_REMOTE_DIR, POD_JOB_CMD, POD_RESULTS, plus optional POD_TMUX_SESSION and POD_LOCAL_ROOT). Copy them into the project, fill in the job command, and commit them next to the code they run.
Pitfalls, in short
Each of these was hit for real. The full catalog, with symptoms and fixes, is references/pitfalls.md.
| Symptom | Cause | Fix |
|---|---|---|
GLIBC_2.38 not found at runtime |
Base image older than the prebuilt binaries copied in | Match the base image release to the binaries' build image; verify with docker run locally |
Binary works in docker run, not over ssh |
sshd login shells do not inherit Docker ENV |
ldconfig for libraries plus /etc/environment for variables |
Permission denied (publickey) on a fresh pod |
REST-created pods get no injected key | Pass the public key as the PUBLIC_KEY env var at create time |
| Launch script dies at exit code 23 after a successful transfer | rsync cannot chown inside the container |
rsync --no-owner --no-group |
| Container never starts, device errors in the log | Host pool with a container-start bug | Pin allowedCudaVersions away from it and build on a matching base |
There are no instances currently available |
Availability is per (type, count) | Query GraphQL stock, then pick another type or count |
| Disk full during dependency install | A tool's requirements.txt pulled a second framework |
One prebaked environment only; audit transitive deps |
| Disk full after weights downloaded | Only one model copy of headroom | Size disk for 2x the model plus caches |
| Fleet launcher reports 1/N launched, rest idle | ssh consumed the loop's stdin |
ssh -n in every loop |
| Every pod fails instantly with the same argparse error | zsh did not word-split the command chain | Generate chains in Python, or run orchestration under bash |
Launch ssh exits 255 with no output, probe works |
pkill -f matched and killed its own session |
Character-class dodge (pkill -f 'serve[r]') or pkill -x, and stop discarding stderr |
| Log empty or frozen while the GPU is busy | Python block-buffers stdout when piped | PYTHONUNBUFFERED=1 on every remote run |
| Monitor calls a healthy pod unreachable | Zero-match grep exit code forwarded by ssh |
; exit 0 on content pipelines; a dedicated probe decides reachability |
Cost discipline
- Write the case for the pod before creating it: what is locally impossible, and what decides that the run is done.
- Give the workload itself a budget or time cap flag as the hard stop, so an overrun ends the work rather than the billing.
pod-down.shruns immediately afterfetch, in the same breath; for a fleet, a local reaper loop makes that automatic (fetch, verify, then delete on chain completion), capping the idle tail of every finished pod at the loop's poll interval.- Record the spend where the work is tracked, with GPU type, count, hourly rate, and wall time.
Additional Resources
- Reference: references/pitfalls.md - the full provisioning pitfall catalog with symptoms, causes, fixes, and what each one cost
- Reference: references/gpu-selection.md - stock queries, the pricing ladder, VRAM per dollar, the single-card to multi-node scale ladder, and the cluster pre-flight checklist
- Reference: references/job-execution.md - detached
tmuxruns, marker discipline, result retrieval, and cost accounting - Reference: references/fanout.md - fanning a grid across a fleet: duration-based slicing, partial-stock queueing, verify-one-then-replicate, retry-with-backoff launchers, idempotent relaunches, fleet monitoring with ETA calibration, and startup anatomy