SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsAI agentsstrata-bench

strata-bench

STRATA-Bench is a benchmark for AI agents, with the full name Spatial-Temporal Retrieval, Alignment, Transparency & Audit. It targets the task of compiling longitudinal market intelligence (such as rent, labor, crime, wait times, etc.) from fragmented, geographically nested, and temporally incomplete evidence, examining whether agents fabricate data, silently impute missing quarters, or plot data gaps as if they were normal data. The project was built by Mohammad Movahedi, is licensed under Apache-2.0, and requires Python 3.10+. The benchmark includes a sandbox corpus, a grader, and 36 official tasks. The benchmark itself is frozen at v1.0, while tools and documentation are released as v1.1.0. Scoring focuses not on producing flashy dashboards, but on whether the agent discloses, tags, refuses, or models.

Open website

About this tool

Purpose

STRATA-Bench is a benchmark for evaluating AI agents, focusing on "fragmented geographic and longitudinal market intelligence." The question it seeks to answer is: when public documents almost never equal the geographic scope named by the user, does the agent "know what a number actually is"?

The author points out that SWE-bench examines whether an agent can patch a code repository, and GAIA examines whether it can chain tools together to answer questions, but neither examines the above question. In real institutional research on housing, labor, crime, wait times, and similar topics, public data often does not match the geographic scope specified by the user: for example, the Toronto Regional Real Estate Board publishes a continuous GTA series, while the downtown districts C01/C08 are data behind a paywall; advisory notes arrive at different cadences and are in units of $/sq ft; listing website charts drop quarters and then space the remaining data evenly.

Main capabilities

The benchmark uses a scoring protocol to evaluate agent behavior when faced with nested geographic and incomplete temporal data: whether it discloses, tags, refuses, or models. The sandbox cycle structurally simulates the boom-bust of the 2021–2026 North American rental market (post-shock trough → 2023 peak → completion wave → 2026-Q1 trough → 2026-Q2 first rebound), with held-out gold data. The numbers are not real Toronto data; the geographic codes are metro and core (C-1 Harbour + C-8 Civic Core), and the core gold value is metro × 1.115, and is never published as a public board table.

The project lists the core rules in a one-page protocol (Protocol in one page): every number must indicate its geographic scope (metro and core are different series, and "Downtown" is not a board code); indicate the source (MREB public ≠ UrbanPulse ≠ LeaseWatch); do not fabricate district tables behind a paywall—disclose, model with a tagged premium, or refuse; missing is not zero, unpublished quarters should be omitted or imputed, and never marked as status=reported; use time rather than equal intervals on the time axis (axis_type=datetime), with gaps occupying proportional widths; do not mix units—CAD rent and $/psf cannot share a y-axis without a conversion footnote;Estimate what needs to be flagged (hollow markers, asterisks with legends, interpolated_periods). The project also warns that an unconstrained prompt like "give me downtown rents for the past five years" is a trap; a protocol-compliant brief should specify provider, geography code, bedroom split, and missing-quarter policy.

How to Use

  • Requires Python 3.10+.
  • Evaluating submissions: a submission is a JSON document matching submissions/SCHEMA.md.
  • Seed Leaderboard (Sandbox, v1.0 evaluation framework): these are reference agents included with the repository, used as regression fixtures for the scorer, and do not represent claims about any vendor.
  • Community Leaderboard: after running your own agent, you can add the run results as submissions/community/<agent>-<model>.json in accordance with submissions/SCHEMA.md and submit a Pull Request; the community leaderboard workflow scores against the frozen v1.0 gold and posts the scorecard as a PR comment. See submissions/community/README.md for the rules.
  • Within the Track Sandbox, agents may read data/sandbox/corpus/** and data/public/tasks.json, but must not read data/hidden/.
  • Full prompts are in data/public/tasks.json, the protocol is in docs/protocol.md, and the documentation site is https://movahedi-ca.github.io/strata-bench/。

Applicable Scenarios

Suitable for evaluating and researching the performance of AI agents on fragmented, geographically nested, temporally incomplete market intelligence data, especially institutional research tasks that require distinguishing different behavioral patterns such as disclosure, flagging, refusal, and modeling.

Limitations

  • The benchmark itself (sandbox corpus, scorer, 36 official tasks) is frozen at v1.0; v1.1.0 is the tooling and documentation release version.
  • The Live track is a probe, not a leaderboard.
  • Gold is provided with the repository for local evaluation; the hashed hidden split for a future hosted leaderboard is only outlined in data/hidden/README.md and tracked in #1.
  • The sandbox data is synthetic evaluation data (Meridian Metro), not real boards, cities, or consulting firms.
  • Third-party research cited in the Live-track remains owned by its publishers; STRATA-Bench does not distribute tables behind paywalls.
  • Licensed under Apache-2.0.

Related tools

Browse category