What is LPM 1.0
LPM 1.0 (Large Performance Model) is a 17B-parameter video character performance generation model launched by Anuttacon (Cai Haoyu's AI company), supporting real-time full-duplex audio-video conversation. The model can transform a single image into adigital humanthat can speak, listen, react, and display subtle micro-expressions, while maintaining identity consistency for unlimited duration. LPM 1.0 serves as a general-purpose visual engine for scenarios such as AI conversation, virtual livestreaming, and game NPCs.
Main Features of LPM 1.0
- Real-time full-duplex conversation: Supports real-time interaction with simultaneous speaking and listening, where both parties can interrupt at any time, and the model can generate natural reactions such as pauses and gaze shifts before responding.
- Unlimited-duration identity consistency: Maintains character appearance, teeth, expression lines, side-face contours, and other details stable throughout hours-long videos based on image input, without the "more generation, more distortion" problem.
- Tri-modal control: Jointly controls character performance through text (controlling actions/expressions), audio (driving lip sync/rhythm), and reference images (maintaining identity).
- Zero-shot generalization: Supports any style including photorealistic humans, 2D anime, 3D game characters, and non-human creatures, without the need for domain-specific fine-tuning.
- Emotional performance: The model can generate subtle micro-expressions such as hesitation, thinking, and breathing rhythm, and supports melody-aligned lip sync during singing.
Technical Principles of LPM 1.0
- Data construction: Through strict quality filtering (retention rate <10%) to remove editing traces, beauty filters, and other defects, an improved LR-ASD model is used to annotate each frame's speaking/listening/idle state and achieve audio separation, while multi-granularity identity reference conditions for global appearance, multi-view body, and facial expressions are constructed, forming a large-scale multimodal dataset.
- Base LPM: Based on a 14B image-to-video pretrained model, 3B parameters of interleaved audio cross-attention blocks are added to form a 17B diffusion Transformer, jointly learning speech-driven dynamics, listening reactions, text control, and multi-reference identity preservation, trained on over 17 trillion multimodal tokens to achieve high-quality character performance generation.
- Online LPM: Through a four-stage autoregressive distillation curriculum, Base LPM is transformed into a causal streaming generator, adopting a Backbone-Refiner architecture to respectively maintain temporal latent trajectories and restore high-fidelity details, achieving low-latency real-time inference and unlimited-length identity-consistent generation.
- System architecture: Plug-and-play compatible with the A2A audio model, cyclically processing the three states of listening, speaking, and idle, and generating the corresponding video stream in real time.
How to Use LPM 1.0
LPM 1.0 is currently only for academic exchange and is not open to the public
LPM 1.0 Project Address
- Project official website：https://large-performance-model.github.io/
- arXiv technical paper：https://arxiv.org/pdf/2604.07823
Key Information and Usage Requirements of LPM 1.0
- Definition: A 17B-parameter video character performance model (Large Performance Model) launched by Anuttacon (Cai Haoyu's AI company), focusing on single-person full-duplex audio-video conversation scenarios, capable of transforming a single image into a digital human that can speak, listen, and react in real time.
- Core capabilities: Real-time full-duplex conversation (supports interruption), unlimited-duration identity consistency (stable appearance/expressions over long periods), tri-modal control (text + audio + image), zero-shot generalization (supports photorealistic/anime/3D/non-human creatures), and subtle emotional performance (micro-expressions/breathing rhythm).
- Technical route: Trained on a multimodal dataset with strict filtering to build the Base LPM (17B diffusion Transformer), distilled into the Online LPM (causal streaming architecture) through four-stage distillation, using a Backbone-Refiner design to achieve real-time generation.
- Application Scenarios: A general-purpose visual engine for conversational agents, virtual livestreaming, game NPCs, AI education tutors, and game companions.
- Current Status：Not publicly available. There are no model weights, source code, online demo, API, or any product; the project page is for academic exchange only.
Core Advantages of LPM 1.0
- Solving the Performance Trilemma: The industry's first video generation model to simultaneously achieve high expressiveness, real-time inference, and long-duration identity stability, breaking through the limitation of traditional models that can only achieve two of these.
- Full-Duplex Real-Time Interaction: Supports true real-time conversation, with seamless switching between speaking and listening states; both parties can speak simultaneously and interrupt at any time, with low response latency and natural micro-reactions such as pauses and gaze shifts.
- Unlimited Duration with Identity Consistency: Through a streaming architecture, details such as character appearance, teeth, and expression lines remain stable over hours-long videos, without the identity drift over time seen in other models (e.g., Kling-Avatar 2.0/OmniHuman 1.5 limited to 30 seconds).
- Natural Listening Behavior: The model can generate authentic listening reactions (nodding, eyebrow movement, gaze), filling the gap left by existing models that focus only on "speaking" while ignoring "listening."
- Zero-Shot Generalization: The model supports any style without fine-tuning, including photorealistic humans, 2D anime, 3D game characters, and non-human creatures, demonstrating extremely strong character adaptation capability.
- SOTA Performance: Leads comprehensively on LPM-Bench, the first interactive character performance benchmark; in human evaluation, the 720P version achieved preference rates of 64.3% and 42.5% over Kling-Avatar-2 and OmniHuman-1.5, respectively.
Comparison of LPM 1.0 with Similar Competitors
| Comparison Dimension | LPM 1.0 | Kling-Avatar 2.0 | OmniHuman-1.5 |
|---|---|---|---|
| Duration Limit | Unlimited duration, long-duration identity stability | Maximum 30 seconds | Maximum 30 seconds |
| Interaction Mode | Full-duplex real-time(can speak/listen/interrupt simultaneously) | One-way speech generation | One-way speech generation |
| Listening Capability | Native support(real-time reactions, nodding, gaze) | Not supported | Not supported |
| Identity Stability | Remains consistent for hours | May drift over time | May drift over time |
| Human Evaluation | Baseline | 64.3%Users prefer LPM | 42.5%Users prefer LPM |
Application Scenarios of LPM 1.0
- Conversational AI Agents: Gives AI assistants a tangible human visual presence, supporting genuine face-to-face interaction, for customer support, virtual assistants, and digital humans.
- Interactive NPCs and Game Characters: Creates open-world NPCs with contextual dialogue, listening behavior, and emotionally coherent body language, enabling interactive storytelling without separate motion capture.
- Livestreaming and Virtual Hosting: Real-time virtual streaming, maintaining identity consistency and visual quality during multi-hour livestreams with sub-second latency, supporting 24-hour round-the-clock broadcasting.
- Education and Personalized Tutoring: AI tutors have a continuous visual presence, can maintain identity consistency during long teaching sessions, and achieve natural transitions from enthusiastic explanation to focused listening.
- Gaming Companion: Real-time AI companions respond to gameplay through contextual commentary, emotional encouragement, and natural expressions, adding a social interaction experience to single-player games.