What is CoF
CoF (Chain-of-Frames) is a new concept introduced by DeepMind, analogous to "Chain-of-Thought" (CoT) in language models. CoF enables video models to reason in time and space, solving complex visual tasks by generating videos frame by frame. For example,Veo 3the model uses CoF to solve maze problems, complete symmetry tasks, or perform simple visual analogy reasoning. Its capability is similar to language models solving problems through symbolic reasoning. CoF achieves visual reasoning by generating coherent video frames, demonstrating the potential of video models in general visual understanding.
Main features of CoF
-
Visual reasoning: By generating videos frame by frame, CoF can solve problems step by step, such as finding a path in a maze, completing symmetry tasks, or performing visual analogy reasoning.
-
Cross spatiotemporal operations: Operate on objects in videos, such as moving, deforming, or changing object attributes, while maintaining the coherence of the video.
-
General visual understanding: CoF helps video models understand physical rules, abstract relationships, and dynamic changes in the visual world, enabling zero-shot learning for general visual tasks.
-
Generate coherent videos: CoF ensures that generated videos are coherent in time and space, enabling the model to generate video content that conforms to logic and physical rules.
Technical principles of CoF
-
Generative models: CoF relies on large-scale generative models. The models are trained on massive data to learn the spatiotemporal structure and dynamic changes of videos.
-
Prompt-driven: Through natural language prompts and initial images, the model is guided to generate videos that meet task requirements. Prompts help the model understand the task goal, and the initial image provides the first frame of the video.
-
Frame-by-frame reasoning: The model generates videos frame by frame, with each step reasoning based on the state of the previous frame and the prompt. The frame-by-frame generation approach is similar to chain-of-thought (CoT) in language models.
-
Physical and logical constraints: Videos generated by CoF need to conform to physical rules and logical consistency. For example, the motion of objects needs to conform to physical laws, and objects in the video cannot violate real-world constraints.
-
Optimization and feedback: Through multiple attempts and optimization, the model can generate more accurate videos. For example, by generating multiple times and selecting the optimal result, the success rate of the task is improved.
CoF project address
- Technical paper：https://papers-pdfs.assets.alphaxiv.org/2509.20328v1.pdf
Application scenarios of CoF
- Maze solving: CoF can generate videos showing how an object finds a path from the start point to the end point in a maze, planning the optimal route frame by frame.
- Visual symmetry tasks: CoF can generate symmetrical patterns or images, completing the drawing of symmetrical figures by filling in blank parts frame by frame.
- Physical simulation: Simulate physical phenomena, such as the motion, collision, and buoyancy of objects, and generate videos that conform to physical laws.
- Image editing: Used in image editing tasks, such as background removal, style transfer, colorization, etc., completing the editing step by step through frame-by-frame video generation.
- Visual analogy: Solve visual analogy problems, for example generating the missing part to complete a visual analogy, and finding the correct solution through frame-by-frame reasoning.