QuantiPhy - A quantitative evaluation benchmark for VLM physical reasoning launched by Fei-Fei Li's team

What is QuantiPhy

QuantiPhy is the first benchmark launched by Fei-Fei Li's team at Stanford University for quantitatively evaluating the physical reasoning ability of vision-language models (VLMs). Through more than 3,300 video-text instances, QuantiPhy requires models to quantitatively reason about the kinematic properties of objects (such as size, speed, and acceleration) based on videos and physical priors (such as object size and speed). Research finds that current VLMs often rely on pretrained knowledge in tasks rather than truly reasoning based on the input, and there is a significant gap between qualitative and quantitative reasoning. QuantiPhy provides an important testing platform for promoting more reliable physical world understanding in VLMs.

Main features of QuantiPhy

• Quantitative evaluation of physical reasoning ability: QuantiPhy focuses on evaluating the quantitative reasoning ability of VLMs regarding the kinematic properties of objects in videos (such as size, speed, and acceleration), filling the gap left by previous qualitative evaluations.
• Provides a standardized testing framework: Through standardized prompts and scoring mechanisms, QuantiPhy can fairly compare the performance of different VLMs and provide a unified measurement standard for models' quantitative physical reasoning ability.
• Reveals the weaknesses of models' reasoning: Through experiments, QuantiPhy reveals the problem that current VLMs rely on pretrained knowledge rather than actual input in quantitative physical reasoning, providing direction for model improvement.
• Supports analysis of diverse scenarios: QuantiPhy's dataset covers two-dimensional and three-dimensional motion, static and dynamic priors, and diverse scene conditions (such as simulation, laboratory, and real world), enabling comprehensive evaluation of models' reasoning ability under different conditions.

Technical principles of QuantiPhy

• Definition of kinematic reasoning tasks: QuantiPhy defines a new task that treats the size, speed, and acceleration of objects as mutually constrained physical quantities. Models need to use given physical priors (such as object length or gravitational acceleration) to recover the world-to-pixel scale and derive other unknown properties through kinematic equations.
• Construction of multi-source datasets: QuantiPhy's dataset consists of simulated, laboratory, and real-world videos, and each source is annotated through physically grounded methods. For example, simulated data directly obtains true physical quantities from Blender simulation parameters, laboratory data calculates physical quantities through multi-camera recording and measured trajectories, and real-world data is annotated by experts based on visual evidence.
• Quantitative evaluation metrics: QuantiPhy uses "Mean Relative Accuracy" (MRA) as the evaluation metric. By measuring the relative error between predicted values and true values, it provides a smoother and more informative evaluation signal for models' quantitative reasoning ability.
• Input faithfulness analysis: QuantiPhy uses controlled experiments (such as removing videos and changing priors) to analyze whether models truly rely on input for reasoning, revealing the current situation that VLMs rely more on pretrained knowledge than input information.

Project address of QuantiPhy

• Project official website：https://quantiphy.stanford.edu/
• GitHub repository：https://github.com/Paulineli/QuantiPhy
• HuggingFace model library：https://huggingface.co/datasets/PaulineLi/QuantiPhy-validation
• arXiv technical paper：https://arxiv.org/pdf/2512.19526

Application scenarios of QuantiPhy

• Autonomous driving and intelligent transportation: Used to evaluate autonomous driving systems' quantitative understanding of the kinematic properties of vehicles and pedestrians, improving traffic scene analysis and accident prevention capabilities.
• Robotics: Helps robots better understand the environment through quantitative physical reasoning, optimizing the precision of operations such as grasping, handling, and navigation.
• Augmented Reality (AR) and Virtual Reality (VR): Enhances the fusion of virtual and real through quantitative reasoning, improving user experience and the realism of interactive scene simulation.
• Industrial Automation: Used for quality inspection and monitoring, optimizing the visual system's quantitative analysis of objects' physical properties to ensure the accuracy and quality of the production process.
• Education and Scientific Research: As a physics education tool and research platform, it helps students and researchers better understand and develop quantitative physical reasoning techniques.