HARNESS-DB is a dataset of LLM agent execution frameworks (agent harnesses). Here, "execution framework" refers to the software between the language model and the task environment, responsible for assembling the model input, executing the actions chosen by the model, and deciding whether to continue. The dataset encodes 1,256 systems from 2022 to 2026 across nine layers and a total of 38 design dimensions, for a total of 47,728 cells.
Each cell either contains a value backed by a verbatim quote and a reopenable locator (usually a file and line number in a specified commit, or a section of a paper), or is explicitly marked as not_reported. The data show that 48.9% of cells are documented silence—the source was read, but the relevant information was not stated. After field weighting, 81.6% of cells in the sandboxing and environment layer are unreported, compared with 24.9% for the control loop layer, indicating that safety-deployment-related layers are the least described. Treating silence as a value would create associations that do not actually exist: of 666 dimension pairs, 265 show an association only when not_reported is treated as a category.
Regarding main capabilities and usage, the DB provides db.cells, a long table (one row per system and dimension), db.wide() (one row per system), db.systems (including stratification and sampling weights), and db.to_hf() (returns a Hugging Face DatasetDict). If not in the cloned directory, hdb.load() can be pointed at the unpacked release, or the environment variable HARNESSDB_DATA can be set. All dimensions for all nine layers and their allowed values are defined in schema/dimensions.json (schema version 1.0.0, frozen); the definition and coding rules for each value are in docs/coding_manual.md, and a mapping to earlier taxonomies is also in the schema.
Data format and access channels: available from GitHub Releases, Zenodo (doi:10.5281/zenodo.23031354; v1.0.0 is 10.5281/zenodo.23031355), and Hugging Face (https://huggingface.co/datasets/bhaskar-ai/harness-db）获取，见 DATASET_CARD.md. It can also be obtained from the cloned repository via python scripts/release_dataset.py --version 1.0.0 --out release/ Build; the build process derives all files from data/systems.json and stops if data validation fails.
Regarding reliability, the coders were model instances working under a fixed protocol (codebook v1.0 and versioned prompts). Of these, 247 systems were independently coded twice through the same pipeline, so the relevant metrics measure the reproducibility of that pipeline (inter-model agreement) rather than accuracy compared with human reading. The intervals are system-level 95% cluster bootstrap intervals (2,000 resamples). Four dimensions have intervals containing the 0.6 lower bound: loop_primitives 0.590 [0.523, 0.657], edit_primitive 0.628 [0.537, 0.711], network_policy 0.668 [0.527, 0.793], and retry_policy 0.676 [0.587, 0.765]. Of these, loop_primitives is the only dimension below the lower bound; it was retained and flagged: it is multivalued, exact set matching counts any additionally listed primitive as a complete mismatch, and its per-value κ is 0.731. No single κ is given for all cells stacked together, because stacking would inflate it. See docs/coding_reliability.md for the relevant reasoning. Checks outside the coding pipeline test coverage rather than cell values: reference set recall was 27/27; the independently built screening pipeline agreed with inclusion decisions at a rate of 0.84 (κ 0.66). To verify the value of a cell, you can open its locator; if the value is wrong, you can report it.
Regarding contribution and collaboration, you can add systems, correct cells, or request that a system be coded. Templates, evidence rules, and validator gates are in CONTRIBUTING.md. docs/COVERAGE_WANTED.md is generated from the data and lists the most-starred systems that have not yet been coded, as well as the dimensions with the sparsest evidence. Open work, from small to large, includes: determining one silent cell for a system you are familiar with (#2, 5–15 minutes); coding an uncoded system (#1); tooling (#6 Dockerfile for the MCP server, #7 loader on PyPI, #9 stale pin checker); research (#3 a larger evidence package, #4 a second human reader, #5 a controlled study varying one agent binding, #8 a comparison table); substantial work can earn co-authorship. Ifyou maintain a framework in the dataset, you can co-maintain its entry. You can communicate in Discussions.
The project uses a preregistered protocol (osf.io/ab2wn; revisions recorded in docs/protocol_prisma_p.md) and a per-cell evidence design: no value enters the dataset without a citation and a locator that readers can reopen. Data, schemas, and documentation are licensed under CC BY 4.0; code (loaders, scripts, validators, browser, CI) is licensed under MIT. CI runs schema freshness checks, validators, and tests on every push and pull request. Every count in the paper is reconciled against the data in docs/count_reconciliation.md.
Limitations and cautions: coding was done by model instances, and the reliability metrics reflect pipeline reproducibility rather than human accuracy; some dimension intervals contain or fall below the 0.6 lower bound; silent cells should not be recoded to a value; assertions about the domain should use sampling weights (coded systems were drawn by stratum from a framework of 6,504 systems, where stratum H is complete and P and O are sampled; the 23 systems coded outside the draw have weight 0), and the loader automatically applies these two rules.
