What is DeepMesh
DeepMesh is a 3D mesh generation framework proposed by researchers from Tsinghua University and Nanyang Technological University, based on reinforcement learning and autoregressive transformers to generate high-quality 3D meshes. It optimizes mesh generation through two key innovations: first, an efficient pretraining strategy that combines a novel tokenization algorithm and an improved data processing pipeline; second, the introduction of reinforcement learning (specifically Direct Preference Optimization, DPO) to align generated meshes with human preferences. DeepMesh can generate meshes with complex details and precise topology based on point cloud and image conditions, outperforming existing methods in both accuracy and quality.
Main Features of DeepMesh
- High-quality 3D mesh generation: DeepMesh can generate 3D meshes with rich details and precise topology, suitable for various complex geometric shapes.
- Point cloud conditional generation: DeepMesh can generate corresponding 3D meshes based on input point cloud data, suitable for various scenarios from sparse point clouds to dense point clouds.
- Image conditional generation: DeepMesh supports image-based conditional generation, and can generate 3D meshes based on input 2D images.
Technical Principles of DeepMesh
- Autoregressive transformer: DeepMesh uses an autoregressive transformer as its core architecture, including self-attention layers and cross-attention layers. It can progressively generate mesh faces, predicting mesh vertices and faces through conditional inputs (such as point clouds or images). For point cloud conditional generation tasks, DeepMesh combines a perceptron encoder to extract point cloud features and integrates them into the transformer model.
- Efficient pretraining strategy: DeepMesh introduces an improved tokenization algorithm that significantly shortens sequence length through locally aware face traversal and block-indexed coordinate encoding while preserving geometric details. The framework adopts data curation and processing improvement strategies, filters low-quality mesh data, and improves training efficiency through a truncation training strategy.
- Reinforcement learning and human preference alignment: DeepMesh introduces Direct Preference Optimization (DPO), designing scoring criteria through human evaluation and 3D metrics, and collecting preference pairs for reinforcement learning training. This makes the generated meshes accurate in geometric precision and more aligned with human aesthetics in visual effects.
- End-to-end differentiable mesh representation: DeepMesh supports end-to-end differentiable mesh representation, and the topology can change dynamically. This differentiability enables the model to be optimized through gradient descent, further improving the quality of generated meshes.
Project Address of DeepMesh
- Project official website：https://zhaorw02.github.io/DeepMesh/
- Github repository：https://github.com/zhaorw02/DeepMesh
- arXiv technical paper：https://arxiv.org/pdf/2503.15265
Application Scenarios of DeepMesh
- Virtual environment construction: DeepMesh can generate realistic 3D mesh models for constructing virtual scenes in virtual reality, such as virtual buildings, virtual cities, etc.
- Dynamic content generation: Through reinforcement learning optimization, DeepMesh can dynamically generate 3D models based on real-time data in games, enhancing the immersion and interactivity of games.
- Character animation: DeepMesh can generate high-quality 3D character models, supporting complex animation production needs such as skeletal rigging and animation rendering.
- Dynamic medical simulation: Through optimization with reinforcement learning, DeepMesh can generate dynamic medical models, such as cardiac motion simulations, helping doctors better understand the movement and function of organs.
- Product Modeling: DeepMesh can be used to generate 3D models of industrial products, supporting complex design and manufacturing processes.