Overview
Foundation-model workloads (protein folding, backbone diffusion, single-cell models) need a GPU and can run for minutes to hours — too long to sit in a synchronous call. This skill is the provider-agnostic dispatch layer the GPU skills build on: a single submit → poll → harvest contract that works the same whether the backend is a SLURM cluster, Modal, RunPod, or GCP. You describe the job once; the dispatcher submits it, returns a handle, polls status to a terminal state, and harvests the output artifacts.
It does not wrap any single model — each model skill (alterlab-alphafold,
alterlab-boltz, alterlab-proteinmpnn, …) describes what to run; this skill describes
where and how to run it.
When to Use This Skill
Use this skill when the user wants to:
- Submit a batch job to a SLURM/HPC cluster and track it to completion (
sbatch→sacct). - Run a GPU job on a managed provider (Modal, RunPod, GCP Batch / Vertex AI) and retrieve results.
- Write a portable job wrapper that runs the same payload across more than one backend.
- Poll a long-running remote job's status and harvest its output files/artifacts.
Does NOT Trigger
| Scenario | Use instead |
|---|---|
| Deploy a serverless container / autoscaling API specifically on Modal | alterlab-modal |
| Actually fold a structure, design a sequence, or run a specific model | the model's own skill (alterlab-alphafold, alterlab-boltz, alterlab-proteinmpnn, …) |
| Local single-machine data analysis with no remote dispatch | the relevant analysis skill (alterlab-scanpy, alterlab-rdkit, …) |
| Query a database over HTTP | the database connector skill (alterlab-pdb, alterlab-uniprot, …) |
The submit → poll → harvest contract
Every backend implements three verbs. Keeping the payload backend-independent is what makes a model skill portable across an HPC allocation and a cloud GPU:
- submit(spec) → handle — enqueue the job; return an opaque handle (SLURM job id, Modal call id, RunPod job id, GCP operation name).
- poll(handle) → status — map the backend's states to a common vocabulary:
PENDING | RUNNING | SUCCEEDED | FAILED | CANCELLED | UNKNOWN. Poll on a backoff; never busy-loop. - harvest(handle) → artifacts — copy the declared output files back to a local
out/directory (scp/rsync from HPC scratch; object-store download for cloud).
scripts/dispatch.py implements submit and poll (plus cancel for SLURM) for the SLURM and a
generic REST backend, and defines the status vocabulary so model skills can depend on it;
harvest is a copy step (rsync/scp or object-store download) shown in references/providers.md. Provider-specific command and
API detail lives in references/providers.md (loaded on demand).
Core Capabilities
1. SLURM / HPC clusters
The classic scheduler path. Submit a job script with resource directives, then poll with
sacct (authoritative for terminal state) rather than only squeue (which drops finished
jobs from its default view):
# Submit; capture the numeric job id from "Submitted batch job <id>"
JOBID=$(sbatch --parsable run.slurm)
# Poll to a terminal state (COMPLETED / FAILED / CANCELLED / TIMEOUT)
sacct -j "$JOBID" --format=State,ExitCode,Elapsed --noheader --parsable2
A minimal GPU job script (run.slurm) — one GPU, an 8-hour wall clock, results on scratch:
#!/bin/bash
#SBATCH --job-name=fold
#SBATCH --gres=gpu:1
#SBATCH --time=08:00:00
#SBATCH --output=%x-%j.out
srun uv run python run_model.py --in input.fasta --out "$SCRATCH/out"
Full directive reference, array jobs, and squeue/scancel usage: see
references/providers.md.
2. Modal — managed serverless GPU
For serverless containers, autoscaling, and .remote() dispatch, this skill defers to
alterlab-modal, which owns the Modal SDK surface. Use Modal when you want zero cluster
management and per-second GPU billing. This skill's role is only to treat a Modal call as one
submit → poll → harvest backend when a workflow needs to stay provider-agnostic.
3. RunPod — on-demand GPU pods
RunPod exposes GPU pods and a serverless endpoint API. Submit to a serverless endpoint and
poll the returned job id; the API key is read from RUNPOD_API_KEY and sent as a bearer token. Endpoint/run/status
paths and the pod vs. serverless trade-off are in references/providers.md.
4. GCP — Batch and Vertex AI
Google Cloud offers Batch (containerized batch jobs with GPU allocation) and Vertex AI
custom jobs (ML-oriented, managed). Both follow submit → poll (operation/job state) → harvest
(read outputs from a GCS bucket). Auth uses Application Default Credentials; the target bucket
comes from an env var. Command/SDK detail: references/providers.md.
5. Portable job wrapper
scripts/dispatch.py is a stdlib-only CLI that runs the same job spec across backends:
# Submit a SLURM job and print the handle
python scripts/dispatch.py submit --backend slurm --script run.slurm
# Poll a handle to a normalized status
python scripts/dispatch.py poll --backend slurm --handle 123456
# Generic REST backend (RunPod-style): endpoint + key from env
python scripts/dispatch.py submit --backend rest \
--endpoint "$RUNPOD_ENDPOINT" --payload spec.json
It shells out to sbatch/sacct for SLURM and uses urllib for the REST backend — no
third-party dependencies, so it runs anywhere Python does.
Validation and status semantics
- Trust the accounting record, not the queue. On SLURM, a job missing from
squeuemay have finished or failed — resolve terminal state withsacct/exit code, not absence. - Poll with backoff (e.g. 10s → 30s → 60s, capped) so you neither hammer the scheduler nor miss a fast job.
- Always check the exit code, not just the state string; a job can report
COMPLETEDwhile the payload wrote no artifacts. Harvest, then verify the expected files exist. - Never hardcode credentials. Read
RUNPOD_API_KEY, GCP ADC, and cluster hosts from the environment; the dispatcher refuses to run if a required secret is unset.
Resources
references/providers.md— per-backend command/API detail (SLURM directives, RunPod endpoints, GCP Batch/Vertex, Modal cross-link) loaded on demand.scripts/dispatch.py— stdlib-onlysubmit/poll/cancelCLI for SLURM + generic REST.
Part of the AlterLab Academic Skills suite.