Make-A-Character: Alibaba's Open-Source AI 3D Digital Human Generation Framework

What is Make-A-Character?

Make-A-Character (Mach for short) is an AI 3D digital human generation framework developed by Alibaba Group's Intelligent Computing Research Institute, designed to quickly create realistic 3D characters from text descriptions. The system is particularly suited to meeting the demand for personalized and expressive 3D characters in AI agents and the metaverse. The core function of Mach is to use large language and vision models to understand intent in text and generate intermediate images, which are then transformed into complete 3D character models through a series of modules targeting human visual perception and 3D generation.

Make-A-Character Official Website Entry

• Official project homepage:https://human3daigc.github.io/MACH/
• Arxiv research paper:https://arxiv.org/abs/2312.15430
• GitHub code repository:https://github.com/Human3DAIGC/Make-A-Character(code to be open-sourced)
• ModelScope Demo：https://www.modelscope.cn/studios/XR-3D/InstructDynamicAvatar/summary

Make-A-Character Features

• Text-to-3D character generation: Users can specify a character's appearance features by inputting text descriptions, such as facial features, hairstyle, clothing, etc., and the Mach system generates the corresponding 3D character model based on these descriptions.
• Flexible controllable customization: The system allows users to customize characters in detail, including facial features, eye shape, iris color, hairstyle and color, eyebrows, mouth and nose, etc., and can even add wrinkles and freckles to create characters that meet users' personalized needs.
• Highly realistic rendering: Mach uses physically based rendering (PBR) technology, combined with real human scan data, to generate highly realistic characters. The character's hair is constructed in the form of hair strands rather than traditional meshes to enhance realism.
• Complete character model: The generated character model includes all details such as eyes, tongue, teeth, full body and clothing, ensuring that the character can be used immediately in various application scenarios.
• Animation support: The character is equipped with advanced skeletal rigs and supports standard animation, enabling the character to perform various dynamic expressions, such as changes in facial expressions.
• Industry compatibility: The character models generated by Mach use explicit 3D representations and can be seamlessly integrated into standard CG pipelines in the game and film industries, facilitating subsequent animation production and rendering.

How Make-A-Character Works

1. Text parsing and visual prompt generation：
   • The user inputs a text prompt describing the character's features.
   • A large language model (LLM) is used to understand the semantic information in the text and extract key facial features and attributes.
   • These features are mapped to visual cues, such as pose and edge maps, to guide subsequent image generation.
2. Reference portrait image generation：
   • Combining the Stable Diffusion model and ControlNet, a reference portrait image is generated based on the extracted visual cues. ControlNet ensures that the generated image has a frontal pose and neutral expression, facilitating subsequent 3D modeling.
   • Openpose and Canny edge detection techniques are used to ensure reasonable distribution of facial features.
3. Dense facial landmark detection：
   • Dense facial landmarks (431 landmark points) are used to reconstruct the facial and head geometry. These landmark points are more detailed than the traditional 68 or 98 points and cover the entire head.
   • Synthetic images are used as training data, and these landmark points are generated through a multi-view capture and processing pipeline.
4. Geometry Generation：
   • Reconstruct the head geometry based on the reference portrait image and dense facial coordinates. Optimize the mesh by mapping the 3D mesh onto a 2D plane.
   • Use coordinate projection loss and local smoothness constraints to ensure the accuracy of the geometric structure.
5. Texture Generation：
   • Use differentiable rendering techniques to extract textures from the reference image, and gradually generate high-resolution textures through a multi-resolution strategy.
   • Introduce a neural de-lighting method to remove unnecessary lighting effects from the texture image, obtaining a diffuse map suitable for rendering.
6. Texture Correction and Completion：
   • Correct the generated diffuse map to address imperfections in areas such as the eyes, mouth, and nostrils.
   • Use a facial parsing algorithm to extract masks for the erroneous regions, and merge them with the template diffuse map using Poisson blending to improve the visual effect.
7. Hair Generation：
   • Synthesize various hairstyles through 2D images, and then perform 3D hair strand reconstruction based on these images.
   • Use advanced hair generation techniques, such as NeuralHDHair, to train models and generate high-quality 3D hair strands.
8. Asset Matching：
   • Match the generated head with pre-made assets such as hair, body, clothing, and accessories.
   • Use the CLIP text encoder to calculate the similarity between the input prompt and asset tags, and select the best-matching asset.
9. Character Rigging：
   • Assemble all generated and matched parts into a complete 3D character model.
   • The character model supports animation and can be dynamically represented through skeletal rigid bodies.