PAST-Bench - A performance attribution benchmark for personal agents launched by Princeton

What is PAST-Bench

PAST-Bench is a benchmark launched by Mengdi Wang's team at Princeton University, used to test the recursive self-improvement capability of personal AI Agents. PAST-Bench determines whether the cross-session experience saved by an Agent truly improves subsequent behavior by comparing task performance under conditions with and without memory. PAST-Bench covers four categories of capabilities: memory, process reuse, information gathering, and updating, with a total of 26 scenarios and 204 task turns.

Main features of PAST-Bench

• Evaluating recursive self-improvement: Testing whether the cross-session experience saved by a personal AI Agent (memory, skills, task history) truly improves subsequent behavior.
• Isolating the effect of experience accumulation: Distinguishing whether "score improvement comes from experience accumulation" or from other factors such as "a stronger base model / Prompt changes / task difficulty."
• Four-dimensional capability diagnosis: Covering the four major dimensions of memory, process reuse, information gathering, and state updating, to pinpoint which specific type of capability an Agent is deficient in.
• Mechanism attribution analysis: Not only looking at the final score, but also tracing the complete path of "storage → retrieval → application" to determine whether the improvement is supported by a real path.

Technical principles of PAST-Bench

• Task family sequence design: The Agent executes multiple rounds of sessions of the same task family in sequence. Early sessions create opportunities to save experience, and later sessions test whether this experience is used correctly.
• Matched control experiment: A control version is designed for each later-round session, with the persistence capability disabled and all other conditions completely identical. The self-evolution difference Δ is obtained through "score with memory – score without memory."
• Mechanism evidence tracking: Telemetry data such as memory writes, skill invocations, session retrieval, and state updates are recorded at runtime, and the Mechanism-Evidence Score is calculated to verify whether the improvement follows the expected path.
• Persistent artifact auditing: Inspecting the content actually saved by the Agent (memory entries, skill files, session indexes), and analyzing its structure, timeliness, and reusability.

How to use PAST-Bench

• Environment preparation: Clone the PAST-Bench repository from GitHub and install the core dependencies and Agent adapter.
• Model configuration: Configure the API Key of the model used in the environment variables.
• Sandbox construction: Execute past-bench build-image --kind sandbox to build the Docker sandbox image required for evaluation.
• Quick verification: Use past-bench evolve to run a single task family and add the --compare-no-persistence parameter for a Smoke Test.
• Full evaluation: Traverse all 26 task families to run the evaluation; each task family automatically executes two sets of control experiments: "with persistence" and "without persistence."
• Result analysis: In the --trace-dir directory, view sequence_comparison.jsonto obtain the Δ value and mechanism evidence score.
• Custom integration: If you need to evaluate your own Agent, implement an adapter in the agents/ directory and ensure that the --compare-no-persistence switch is supported.

Core advantages of PAST-Bench

• True attribution capability: PAST-Bench can precisely distinguish whether improvements come from experience accumulation, the model itself getting stronger, prompt changes, or the task becoming simpler.
• Matched controlled experiment design: Each task family is paired with a control version with persistence disabled, with all other conditions identical, enabling a direct causal comparison between with-memory and without-memory.
• Mechanism-level diagnosis: Proposes the Mechanism-Evidence Score, determining whether the Agent answers correctly, and also traces the complete path of "storage -> retrieval -> application", identifying the difference between "answering correctly by chance" and "truly reusing experience".
• Four-dimensional capability decomposition: Splits the vague notion of self-improvement into four specific capabilities: memory, process reuse, information gathering, and state updating, precisely locating weaknesses.
• Diagnosis-driven improvement: Based on runtime failures exposed by the benchmark, it directly led to the Hermes+ framework, proving that it is not only an evaluation tool but also a diagnostic engine for Agent architecture optimization.

Project address of PAST-Bench

• GitHub repository：https://github.com/Gen-Verse/PAST-Bench
• arXiv technical paper：https://arxiv.org/pdf/2608.04003

Comparison of PAST-Bench with similar competitors

Application scenarios of PAST-Bench

• Academic research: Provides a standardized, reproducible quantitative evaluation environment for Agent recursive self-improvement, supporting objective comparison of different architectures.
• Framework development: Precisely locates weaknesses in Agent frameworks through four-dimensional capability decomposition, guiding architecture optimization (such as the birth of Hermes+).
• Model selection: Compares the cross-session experience reuse performance of different base models under a fixed framework, identifying each model's capability tendencies.
• Persistence verification: Test the actual effects of different memory structures and retrieval strategies, avoiding ineffective persistence where something is "stored but cannot be found" or "stored but useless."
• Product iteration: Distinguish whether the score improvement of a new version comes from "cross-session experience accumulation" or from "a stronger base model," ensuring that the persistence feature truly delivers value.