SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

Sitcom-Crafter - A 3D Human Motion Generation System Jointly Launched by Beihang University, CUHK, and Other Universities

Author: AI小集 · Original source: AI工具集

What is Sitcom-Crafter

Sitcom-Crafter is a 3D human motion generation system jointly launched by Beihang University, The Chinese University of Hong Kong (Shenzhen), University of Technology Sydney, Sun Yat-sen University, and other universities. Based on long storyline instructions provided by users, it generates diverse and physically realistic motions, including human walking, human-scene interaction, and human-human interaction. The system comprises eight modules, among which three core modules are responsible for motion generation, and five enhancement modules are used to optimize motion smoothness, synchronization, collision correction, etc. Sitcom-Crafter's innovative 3D scene-aware technology and self-supervised SDF strategy generate physically logical motions without additional data collection, significantly improving creation efficiency and providing an efficient and intelligent solution for animation and game design.

Sitcom-Crafter

Main Features of Sitcom-Crafter

  • Human Walking: Characters walk naturally in 3D scenes.
  • Human-Scene Interaction: Characters interact with environmental objects in a physically logical manner.
  • Human-Human Interaction: Coordinated motion generation between characters, reducing collisions and optimizing motion synchronization.
  • Enhancement Features：
    • Storyline Parsing: AI automatically breaks down scripts and converts them into specific character motion instructions.
    • Motion Synchronization: Ensures that motions generated by different modules remain temporally consistent.
    • Hand Pose Enhancement: Enhances the naturalness of motion by retrieving hand poses from a database.
    • Collision Correction: Automatically detects and corrects collisions between characters.
    • 3D Retargeting: Maps generated motions onto existing 3D digital character models to improve visual effects.

Technical Principles of Sitcom-Crafter

  •  Motion Generation Modules：
    • Human Walking Generation: Based on existing advanced methods (such as GAMMA), generates character walking motions through deep learning models.
    • Human-Scene Interaction Generation: Based on methods such as DIMOS, combines scene information to generate interaction motions between characters and scene objects.
    • Human-Human Interaction Generation: Uses a self-supervised scene-aware method to synthesize SDF (Signed Distance Function) points in 3D scenes to simulate the surrounding environment and avoid collisions between characters and the scene. Based on a diffusion model, generates multi-person interactions, considering the relative positions and motions between characters.
  • Scene-Aware Technology：
    • SDF Point Synthesis: Based on preprocessed motion data, defines motion regions and randomly generates SDF points of surrounding objects to simulate obstacles in the scene.
    • Collision Detection and Correction: SDF points detect collisions between characters and the scene or each other, and adjust motion trajectories or speeds to avoid collisions.
  • Storyline Understanding and Command Generation: Uses large language models (such as Gemini 1.5) to parse storyline text and convert it into specific motion instructions. Based on natural language processing technology, decomposes complex storylines into multiple motion commands and assigns them to the corresponding motion generation modules.
  • Enhancement Modules：
    • Motion Synchronization: Based on interpolation techniques (such as Slerp), ensures smooth temporal transitions between motions generated by different modules.
    • Hand pose retrieval: The CLIP model retrieves the hand pose most similar to the text description and incorporates it into the generated motion.
    • Collision correction: Adjust the motion speed or path to avoid collisions between characters.
    • 3D retargeting: Map the generated motion onto a high-quality 3D digital character model to improve the visual effect.
  • Unified motion representation: The system uses marker points as a unified motion representation, supporting the extraction of motion data from different data sources (such as SMPL and SMPL-X models), enhancing the system's scalability and compatibility.

Sitcom-Crafter project address

Application scenarios of Sitcom-Crafter

  • Animation production: Quickly generate character motion, reduce manual animation workload, and support diverse actions and plot-driven animation design.
  • Game development: Design natural behaviors and interactive actions for NPCs, support dynamic plot generation and real-time action feedback, and enhance game immersion.
  • Virtual reality (VR) and augmented reality (AR): Generate natural interactions and scene simulations for virtual characters, enhancing user experience and interaction naturalness.
  • Film and television production: Used for early creative validation, special effects scene design, and motion capture replacement, improving production efficiency.
  • Education and training: Generate character behaviors in simulated training, create virtual teaching assistants, and support safe demonstrations and complex scenario teaching.