What is CAD-MLLM
CAD-MLLM is a computer-aided design (CAD) model generation system jointly launched by ShanghaiTech University, Transcengram, DeepSeek AI, and the University of Hong Kong. It generates parametric CAD models based on various user inputs, such as text descriptions, images, point clouds, or combinations of these inputs. The system uses command sequences and large language models (LLMs) to align and process multimodal data, building complete CAD models. CAD-MLLM introduces a large-scale multimodal dataset called Omni-CAD, as well as new evaluation metrics to comprehensively assess the topological quality and surface closure degree of generated models. CAD-MLLM surpasses existing methods in performance and demonstrates high robustness to data defects.
Main features of CAD-MLLM
- Multimodal input processing: Processes various input forms including text descriptions, images, and point clouds, and generates CAD models based on the input.
- Parametric CAD model generation: The system can generate parametric CAD models, and users can edit and adjust the generated models.
- Dataset construction and annotation: Introduces a dataset named Omni-CAD, containing text descriptions, multi-view images, point clouds, and corresponding CAD command sequences.
- Innovative evaluation metrics: The system introduces new evaluation metrics to assess the topological quality and surface closure degree of generated CAD models.
- Robustness: Demonstrates high robustness when handling noise and missing data.
- Interactive design: Users can easily design CAD models based on simple instructions and illustrations, allowing non-experts to realize design ideas.
Technical principles of CAD-MLLM
- Command sequence representation: Uses command sequences of CAD models, vectorizes the command sequences, and forms a data stream suitable for large language model (LLM) learning.
- Multimodal data alignment: Uses advanced LLMs to align vector representations of different modal data and CAD models, enabling the model to understand and process multiple inputs.
- Network architecture: The network architecture includes three modules: visual data alignment, point data alignment, and large language model, supporting cross-modal input.
- Feature space sharing: Non-text input is first processed based on frozen encoders, and then a projection layer is used to align features within the shared large language model (LLM) feature space.
- Low-rank adaptation (LoRA) fine-tuning: Based on integrating prompts with multimodal embeddings, low-rank adaptation (LoRA) technology is applied to fine-tune the LLM to generate accurate CAD models.
- Data augmentation methods: Proposes data annotation processes and data augmentation methods to generate a new multimodal conditional CAD dataset Omni-CAD.
Project addresses of CAD-MLLM
- Project official website：cad-mllm.github.io
- arXiv technical paper：https://arxiv.org/pdf/2411.04954
Application scenarios of CAD-MLLM
- Industrial design and manufacturing: Designers and engineers can quickly generate and modify complex industrial product CAD models, accelerating the product development process.
- Architecture and engineering: Architects and structural engineers can generate precise CAD drawings from on-site photos or terrain data, improving the efficiency of design and planning.
- Automotive industry: Automakers can generate precise CAD models of automotive components from concept sketches or descriptions, optimizing design and manufacturing processes.
- Aerospace: In the field of aerospace, generate CAD models of aircraft and spacecraft components and structures from complex design requirements and performance parameters.
- Education and training: For students and beginners, lower the learning curve and improve teaching effectiveness.