What is ENEL
ENEL (Exploring the Potential of Encoder-free Architectures in 3D LMMs) is an innovative encoder-free 3D Large Multimodal Model (3D LMM) that addresses the limitations of traditional encoder architectures in 3D understanding tasks. By removing the 3D encoder, ENEL directly converts point cloud data into discrete point tokens, which are concatenated with text tokens and fed into a Large Language Model (LLM). The model achieves efficient semantic encoding and geometric structure understanding through two key strategies: first, the LLM-embedded semantic encoding strategy, which extracts high-level semantics through a hybrid semantic loss; second, the hierarchical geometry aggregation strategy, which enables the LLM to attend to local details of the point cloud.
ENEL's 7B model performs excellently on multiple 3D tasks, including 3D object classification, 3D object captioning, and 3D visual question answering (VQA). On the Objaverse benchmark, ENEL-7B achieves a GPT score of 50.92% on the captioning task and 55.0% on the classification task, and reaches 42.7% on the VQA task of the 3D MM-Vet dataset, all comparable to existing 13B models (such as ShapeLLM). ENEL's encoder-free architecture performs excellently in semantic encoding and can better capture the semantic correlation between point clouds and text.
Main Features of ENEL
- Encoder-free Architecture: ENEL removes the traditional 3D encoder and directly converts point cloud data into discrete point tokens through a token embedding module, which are concatenated with text tokens and fed into the LLM. It avoids the common issues in encoder architectures such as point cloud resolution limitations and semantic embedding mismatch.
- High-level Semantic Extraction: Through the LLM-embedded semantic encoding strategy, ENEL introduces a Hybrid Semantic Loss during the pretraining stage, which can extract high-level semantic features of point clouds while preserving key geometric structures.
- Local Geometry Awareness: During the instruction tuning stage, ENEL adopts a hierarchical geometry aggregation strategy, enabling the LLM to actively perceive local details of the point cloud. Through aggregation and propagation operations, local geometric information is integrated into the early layers of the LLM, achieving fine-grained understanding of complex 3D structures.
- Multi-task 3D Understanding: ENEL performs excellently on multiple 3D tasks, including 3D object classification, 3D object captioning, and 3D visual question answering (VQA). The 7B model achieves 55.0% classification accuracy and a 50.92% captioning GPT score on the Objaverse benchmark, with performance comparable to the 13B ShapeLLM.
- Efficient Semantic Alignment: ENEL achieves efficient semantic alignment between point cloud and text modalities through its encoder-free architecture, better capturing the semantic correlation between the two and providing a stronger semantic foundation for 3D multimodal tasks.
Technical Principles of ENEL
- LLM-embedded Semantic Encoding: During the pretraining stage, ENEL explores different point cloud self-supervised losses (such as masked modeling loss, reconstruction loss, contrastive loss, and knowledge distillation loss) and proposes a Hybrid Semantic Loss. This loss function can embed high-level semantic information of point clouds into the LLM, replacing the function of traditional 3D encoders.
- Hierarchical Geometry Aggregation: During the instruction tuning stage, ENEL introduces a hierarchical geometry aggregation strategy. This strategy incorporates inductive bias into the LLM by performing aggregation and propagation operations on point clouds in the early layers of the LLM, enabling attention to local details of the point cloud. Specifically, it uses farthest point sampling (FPS) and k-nearest neighbors (k-NN) algorithms to downsample and aggregate the point cloud, progressively integrating fine-grained semantic information of the point cloud.
Project Address of ENEL
- Github repository：https://github.com/Ivan-Tang-3D/ENEL
- arXiv technical paper：https://arxiv.org/pdf/2502.09620
Application scenarios of ENEL
- 3D object classification: ENEL can efficiently classify 3D objects, making it suitable for fields such as industrial automation, robot vision, and autonomous driving, and it can quickly recognize and classify complex 3D objects.
- 3D object caption generation:It can be used to generate descriptive text for 3D models, helping users quickly understand key information in 3D scenes, and is suitable for virtual reality (VR), augmented reality (AR), and 3D modeling tools.
- 3D visual question answering (VQA): It can answer questions related to 3D scenes, for example in medical imaging analysis, architectural design, and the education field, helping users quickly obtain key information from 3D data.
- Understanding of complex 3D geometric structures: It is suitable for scenarios that require precise understanding of complex geometric shapes, such as aerospace, automotive manufacturing, and jewelry design, and can help engineers and designers better understand and optimize 3D models.