What is Find3D
Find3D is a 3D part segmentation model launched by Caltech that can segment any part of any object based on arbitrary text queries. Find3D uses a powerful data engine to automatically generate training data from 3D assets on the internet, and uses a contrastive training method to train a scalable 3D model. Find3D demonstrates excellent performance on multiple datasets including Objaverse-General, ShapeNet-Part, and PartNet-E, achieving up to a 3x improvement in mean Intersection over Union (mIoU) over the next-best method, and can handle in-the-wild 3D reconstructions from iPhone photos and AI-generated images.
Main features of Find3D
- Open-world 3D part segmentation: Identify and segment any part of any object simply with a text query, without being limited to a predefined set of parts.
- No manual annotation required: Uses a data engine to automatically generate training data from 3D assets on the internet, without manual annotation.
- High performance and generalization ability: Performs excellently on multiple datasets, with a 3x improvement in mean Intersection over Union (mIoU) compared with the next-best method.
- Fast inference: 6 to 300 times faster than existing baselines, significantly improving inference speed.
- Robustness: Maintains stable segmentation results under different object poses and query conditions.
- Query flexibility: Supports different types of text queries, including part queries with different granularities and description styles.
Technical principles of Find3D
- Data engine：
- Uses 2D foundation models (such as SAM and Gemini) to automatically annotate 3D objects.
- Renders 3D assets into multiple views, and passes each view to SAM for segmentation.
- For each mask returned by SAM, queries Gemini to obtain the corresponding part name, forming (mask, text) pairs.
- Embeds part names into the latent embedding space of vision and language foundation models (such as SigLIP).
- Based on projective geometry, back-projects the masks into the 3D point cloud to form (point, text embedding) pairs.
- Model training：
- A Transformer-based point cloud model that treats the point cloud as a sequence and performs block attention.
- Queries any free-form text by the cosine similarity between the point features returned by the model and the text embeddings.
- Uses a contrastive learning objective to handle label polysemy and partial visibility issues, supporting scalable training on data generated by the data engine.
- Contrastive learning objective：
- Solves the problem that each point has multiple labels, as well as the problem of unlabeled points caused by each mask covering only part of the views.
- Based on the contrastive learning objective, enables the model to learn robust feature representations and achieve accurate part segmentation in the open world.
Project address of Find3D
- Project website：ziqi-ma.github.io/find3dsite
- GitHub repository：https://github.com/ziqi-ma/Find3D
- arXiv technical paper：https://arxiv.org/pdf/2411.13550v1
- Online demo：https://huggingface.co/spaces/ziqima/Find3D
Application scenarios of Find3D
- Robot vision and manipulation：In the robotics field, it helps robots identify and locate specific parts of objects for precise grasping, manipulation, or interaction.
- Virtual reality (VR) and augmented reality (AR)：In VR/AR applications, it provides a deeper understanding of virtual objects and enhances the user's interactive experience with the virtual environment.
- Computer-Aided Design (CAD)：In CAD software, it helps designers quickly identify and edit specific parts of 3D models, improving design efficiency.
- Game Development：In game development, create more complex 3D object interactions, such as changing a character's equipment or object destruction effects.
- Architecture and Engineering：In the fields of architecture and engineering, it helps analyze and understand complex 3D structures, such as architectural models or mechanical components.