What is See3D
See3D (See Video, Get 3D) is a 3D generation model launched by the Beijing Academy of Artificial Intelligence, capable of learning from large-scale unlabeled internet videos to generate 3D content from videos. Unlike traditional 3D generation models that rely on camera parameters, See3D adopts visual conditioning technology, generating multi-view images with controllable camera directions and geometric consistency solely through visual cues in videos. It avoids the need for expensive 3D or camera annotations and can efficiently learn 3D priors from internet videos. See3D supports generation from text, single view, and sparse views to 3D, and can perform 3D editing and Gaussian rendering.
Main Features of See3D
- Generation from Text, Single View, and Sparse Views to 3D: See3D can generate 3D content based on text descriptions, a single-view image, or a small number of images.
- 3D Editing and Gaussian Rendering: The model supports editing the generated 3D content and uses Gaussian rendering technology to improve rendering effects.
- Unlocking 3D Interactive Worlds: After inputting an image, immersive and interactive 3D scenes can be generated, supporting users in exploring real spatial structures in real time.
- 3D Reconstruction Based on Sparse Images: With a small number of input images (3-6), the model can generate refined 3D scenes.
- Open-World 3D Generation: Based on text prompts, the model can generate artistic images and, based on these images, generate virtualized 3D scenes.
- Single-View-Based 3D Generation: With a single input image of a real scene, the model can generate realistic 3D scenes.
Technical Principles of See3D
- Visual Conditioning Technology: See3D does not rely on traditional camera parameters and adopts visual conditioning technology, generating multi-view images with controllable camera directions and geometric consistency through visual cues in videos.
- Large-Scale Unlabeled Video Learning: See3D can efficiently learn 3D priors from internet videos without relying on expensive 3D or camera annotations.
- Dataset Construction: The team constructed a high-quality, diverse, large-scale multi-view image dataset WebVi3D, covering 320 million frames from 16 million video clips. The dataset can be continuously expanded through an automated process as the volume of internet videos grows.
- Multi-View Diffusion Model Training: See3D introduces a new visual condition, generating pure 2D inductive visual signals by adding time-dependent noise to masked video data, supporting scalable multi-view diffusion model (MVD) training, avoiding dependence on camera conditions, and achieving the goal of 'obtaining 3D through vision alone'.
- 3D Generation Framework: The 3D priors learned by See3D enable a series of 3D creation applications, including single-view-based 3D generation, sparse-view reconstruction, and 3D editing in open-world scenes, supporting the generation of long-sequence views under complex camera trajectories at both the object level and scene level.
See3D Project Address
- Project Official Website：https://vision.baai.ac.cn/see3d
- Github Repository：https://github.com/baaivision/See3D
- arXiv Technical Paper：https://arxiv.org/pdf/2412.06699
Application Scenarios of See3D
- Game Development: AI-generated 3D models can be used to create characters, environments, and objects in games, improving development efficiency and reducing costs.
- Architectural Design: In architectural design, AI can generate building models, helping designers quickly conceive and modify design plans.
- E-commerce: Online retail can use AI-generated 3D models for product displays, improving the user shopping experience.
- AR/VR: In the AR/VR field, AI-generated 3D models can be used to create realistic virtual environments and characters, enhancing the user's sense of immersion.
- Film and entertainment: AI can help filmmakers create CG characters by replacing live-action characters, simplifying the visual effects production process.
- Industrial design: AI-generated 3D models can be used to simulate the design of industrial products, accelerating the product development process.