What is Matrix3D
Matrix3D is a new unified photogrammetry model jointly launched by Nanjing University, Apple, and The Hong Kong University of Science and Technology. It can complete multiple photogrammetry subtasks in a single model, including pose estimation, depth prediction, and novel view synthesis. The core of Matrix3D is a multimodal Diffusion Transformer (DiT), which integrates data from multiple modalities such as images, camera parameters, and depth maps to enable flexible task processing. Matrix3D is trained using a masked learning strategy, and even when some data is missing, it can still perform full-modality training based on bimodal data (such as image-pose or image-depth pairs), significantly increasing the amount of available training data. Matrix3D supports multi-round interaction, allowing users to progressively input information to refine generation results, offering strong flexibility in 3D content creation.
Main Features of Matrix3D
- Pose Estimation: It can estimate camera poses from images with sparse viewpoints and accurately predict the camera's relative position and orientation even when image overlap is low.
- Depth Prediction: It can predict high-quality depth maps from monocular or multi-view images. The model supports generating depth information from a small number of images, which can be used for subsequent 3D reconstruction tasks.
- Novel View Synthesis: It can generate novel view images from any pose based on input images. It supports synthesizing new viewpoints from a single image or multiple images.
- 3D Reconstruction: By combining the results of pose estimation, depth prediction, and novel view synthesis, Matrix3D can be used for 3D reconstruction from a single image or a small number of images. By integrating with 3D Gaussian Splatting (3DGS) optimization, the model can generate high-quality 3D point clouds.
- Multi-task Interaction and Flexibility: Matrix3D supports flexible input/output configurations and can dynamically adjust tasks based on the information provided by the user.
- Masked Learning and Data Utilization: Through the masked learning strategy, Matrix3D can be trained on partially complete data, significantly expanding the range of available training data and improving the model's generalization ability.
Technical Principles of Matrix3D
- Multimodal Diffusion Transformer: Matrix3D uses a Diffusion Transformer architecture that can process data from multiple modalities (such as RGB images, camera poses, and depth maps) and unify them into a two-dimensional representation, enabling cross-modal feature fusion and generation.
- Masked Learning Strategy: During training, the model adopts masked learning, randomly masking input data and learning to recover clean outputs from noisy data. It can handle incomplete input data and significantly increases the amount of available training data.
- Unified Probabilistic Model: Matrix3D is based on a unified probabilistic model, and through flexible input/output configurations, it can dynamically adjust task requirements, supporting multiple photogrammetry tasks such as pose estimation, depth prediction, and novel view synthesis.
- 3D Gaussian Splatting Optimization: The generated depth maps and novel view images can be used to initialize 3D Gaussian Splatting (3DGS) optimization, achieving high-quality 3D point cloud reconstruction.
- Data Normalization: Through scene normalization and camera normalization techniques, Matrix3D can handle the diverse distributions of different datasets and ensure that data from different modalities are processed under a unified scale and coordinate system.
Matrix3D Project Address
- Project Website：https://nju-3dv.github.io/projects/matrix3d/
- arXiv Technical Paper：https://arxiv.org/pdf/2502.07685
Application Scenarios of Matrix3D
- Virtual Reality (VR) and Augmented Reality (AR): Matrix3D can generate high-quality novel views and depth information from a small number of 2D images, providing rich 3D scenes and object models for VR and AR applications.
- Game Development: Matrix3D can quickly generate geometric information for 3D scenes and objects, helping game developers rapidly create 3D models in games from concept art or a small number of reference images.
- Film and Television Production: Matrix3D can be used to generate complete 3D scenes from a small number of captured 2D images, facilitating post-production visual effects and virtual cinematography.
- Architectural Design: Matrix3D can generate 3D architectural models from architectural drawings or a small number of photos, helping designers quickly conduct design validation and client presentations.
- Virtual Try-On: Matrix3D can generate 3D models of users' bodies for virtual try-on of clothing and accessories.