Purpose Shap-E is the official code and model released by OpenAI for generating 3D objects conditioned on text or images, and the related paper is "Shap-E: Generating Conditional 3D Implicit Functions."
Main capabilities
- Text-conditioned 3D model generation: use the sample_text_to_3d.ipynb notebook to sample and generate 3D models from text prompts.
- Image-conditioned 3D model generation: use the sample_image_to_3d.ipynb notebook to sample and generate 3D models from synthetic view images. For the best results, the background of the input image should first be removed.
- Model encoding and reconstruction: use encode_model.ipynb to load a 3D model or trimesh, generate a batch of multi-view renderings and point clouds, encode them into a latent representation, and then render them back. This feature requires Blender 3.3.1 or higher to be installed and the environment variable BLENDER_PATH to be set to the path of the Blender executable.
- Example display: the repository provides examples of what the text-conditioned model can generate and lists random samples on selected prompts (see samples.md).
How to use The installation command is pip install -e . . Then you can get started by referring to the three example notebooks above.
Applicable scenarios Suitable for experimentation and development in generating 3D objects from text or images.
Limitations When using image-conditioned generation, it is recommended to first remove the background of the input image; the model encoding feature depends on Blender 3.3.1 or higher and the correct BLENDER_PATH environment variable. Other limitations not explained in the materials are not provided for now.
