SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

SoulX-LiveAct - Soul App's open-source real-time digital human generation framework

Author: AI小集 · Original source: AI工具集

What is SoulX-LiveAct

SoulX-LiveAct is an open-source real-timedigital human generation frameworkfrom the Soul App AI team, addressing the stability challenges of streaming generation in AR diffusion models. Core innovations include: Neighbor Forcing technology aligns the diffusion steps of adjacent frames to ensure visual consistency; the ConvKV Memory mechanism achieves constant VRAM usage, supporting hour-level or even unlimited-duration generation. With just dual H100/H200 GPUs, it can achieve 20 FPS real-time inference with only 0.94 seconds of latency. SoulX-LiveAct is suitable for scenarios such as live streaming, virtual customer service, and podcasts, marking a new stage where open-source digital human technology enters a deployable production environment.

SoulX-LiveAct

SoulX-LiveAct's main features

  • Real-time portrait animation generation: Generates high-fidelity digital human videos in real time based on audio and text conditions, achieving precise lip sync, natural facial expressions, and coordinated body movements.
  • Hour-level/unlimited-duration video: Breaks through the duration limitations of traditional models through a constant VRAM mechanism, stably generating hour-level or even unlimited-duration continuous video streams.
  • Emotion and motion controllable editing: Supports flexible control of head pose, hand gestures, and facial expressions through text instructions, such as heart-shaped gestures, covering the face, and laughing, while maintaining identity consistency and accurate lip sync.
  • Low-latency streaming inference: With just dual H100/H200 GPUs, it achieves 20 FPS real-time output with an end-to-end latency of only 0.94 seconds, meeting the needs of real-time interactive scenarios such as live streaming and virtual customer service.

SoulX-LiveAct's technical principles

  • Neighbor Forcing: Traditional AR diffusion models use different diffusion steps for adjacent frames, leading to distribution inconsistency and visual jitter. Neighbor Forcing enforces that adjacent frames are generated under the same diffusion step, using the latent variables of the previous frame as conditional input for the current frame, placing the generation process in a consistent noise space, eliminating cross-step alignment issues, and achieving stable temporal coherence.
  • ConvKV Memory: The VRAM bottleneck in long video generation stems from the KV cache growing linearly with the number of frames. ConvKV Memory adopts a "short-term precise + long-term compressed" strategy: retaining high-precision KV cache for recent frames to ensure coherence, compressing historical frames into fixed-length memory through 1D convolution (compression ratio 5:1), and resetting RoPE positional encoding.
  • End-to-end performance optimization: The system adopts adaptive FP8 precision to reduce computational load, combined with sequence parallelism to fully utilize multi-GPU compute power, and reduces memory access overhead through operator fusion. These three measures together achieve 20 FPS real-time inference, requiring only 27.2 TFLOPs per frame, reducing computational cost by 30%-45% compared to similar methods.

SoulX-LiveAct's key information and usage requirements

  • Project positioning: A real-time interactive digital human generation framework open-sourced by Soul App AI Lab, addressing the stability and duration limitations of AR diffusion models in streaming generation, and supporting hour-level or even unlimited-duration video synthesis.
  • Core breakthrough – Neighbor Forcing: Aligns adjacent frames with the same diffusion step, eliminating visual jitter caused by distribution inconsistency.
  • Core breakthrough – ConvKV Memory: Constant VRAM usage, breaking through the duration bottleneck.
  • Core breakthrough – Real-time performance: 20 FPS streaming inference, 0.94 seconds latency.
  • Measured performance – Resolution: 512×512 or 720×416.
  • Measured performance – Frame rate：20 FPS。
  • Measured performance – Latency: 0.94 seconds.
  • Measured performance – Computational cost: 27.2 TFLOPs/frame.
  • Recommended Configuration – GPU: 2× NVIDIA H100 or H200.
  • Recommended Configuration – Environment: Python 3.10, CUDA support.
  • Recommended Configuration – Key Dependencies: SageAttention (FP8 attention), vLLM (FP8 GEMM), LightVAE.
  • Consumer-grade GPUs – Applicable Models: RTX 4090/5090 single card.

Core Advantages of SoulX-LiveAct

  • Neighbor Forcing Technology: Aligns adjacent frames through the same diffusion step, eliminating画面 jitter caused by distribution inconsistency in traditional AR diffusion models, ensuring a stable and coherent generation process.
  • ConvKV Memory Mechanism: Adopts a "short-term precise + long-term compressed" strategy, compressing historical KV caches into a fixed length to achieve constant VRAM usage, breaking through the duration bottleneck, and supporting hour-level or even unlimited-length generation.
  • Real-time Streaming Inference: The model only requires dual H100/H200 cards to achieve 20 FPS real-time output, with an end-to-end latency of only 0.94 seconds, meeting the needs of real-time interactive scenarios such as live streaming.
  • Low Computational Cost: Each frame requires only 27.2 TFLOPs, reducing computational cost by 30%-45% compared to similar methods, balancing high quality and high efficiency.
  • Long-term Consistency: Maintains stable character identity, no loss of key details, and precise lip synchronization in hour-level videos, avoiding issues such as identity drift and accessories flickering in and out.

How to Use SoulX-LiveAct

  • Environment Preparation: Use conda to create and activate a Python 3.10 environment named liveact.
  • Install Basic Dependencies: Install the dependencies in requirements.txt via pip, and install the sox audio processing tool via conda.
  • Install SageAttention: Clone the SageAttention repository and switch to version v2.2.0, then run setup.py to install and enable FP8 attention acceleration.
  • Install the QKV Operator Fusion Version (Optional): Clone the SageAttentionFusion repository and install it to further improve operator fusion performance.
  • Install vLLM: Install vLLM version 0.11.0 via pip to provide FP8 GEMM matrix computation support.
  • Install LightVAE: Clone the LightX2V repository and run setup_vae.py to install the video encoding and decoding components.
  • Download Model Weights: Download the SoulX-LiveAct model files from Hugging Face or ModelScope to a local directory.
  • Download Audio Encoder: Obtain the chinese-wav2vec2-base audio feature extraction model.
  • Dual-card H100/H200 Real-time Inference: Set environment variables and run torchrun to start dual-card distributed inference, specifying the model path, audio encoder path, and input JSON file, and enabling 20 FPS streaming audio generation.
  • Inference Supporting Motion/Expression Editing: Use 512×512 resolution and 24 FPS frame rate, and load the example_edit.json file containing editing instructions to achieve controllable expression and motion generation.
  • Running on RTX 4090/5090 Consumer-grade GPUs: In single-card mode, enable FP8 KV cache, VRAM block offloading, and T5 text encoder CPU offload to reduce VRAM usage and run on consumer-grade GPUs.
  • Prepare Input Data: Edit the JSON configuration file to specify generation parameters such as the reference image path, driving audio path, and emotion and motion text prompts.
  • Start Real-Time Streaming Generation: After executing the inference command, the system outputs in real time, based on the audio input, a digital human video stream with synchronized lip movements and coordinated facial expressions and motions.

SoulX-LiveAct Project Links

  • Project Website：https://soul-ailab.github.io/soulx-liveact/
  • GitHub Repository：https://github.com/Soul-AILab/SoulX-LiveAct
  • HuggingFace Model Hub：https://huggingface.co/Soul-AILab/LiveAct
  • arXiv Technical Paper：https://arxiv.org/pdf/2603.11746

Comparison of SoulX-LiveAct with Similar Competing Products

Comparison Dimension InfiniteTalk Live-Avatar OmniAvatar SoulX-LiveAct
Inference Efficiency
Throughput 25 FPS 20 FPS – 20 FPS
Latency 3.20 s 2.89 s – 0.94 s
Number of GPUs 8 5 – 2
TFLOPs per Frame 50.2 39.1 – 27.2
Long-Duration Generation Capability
VRAM Usage Linear growth Linear growth Linear growth Constant
Maximum Duration Limited by VRAM Limited by VRAM Limited by VRAM Unlimited
Identity Consistency Drifts in later stages Gradual drift Severe drift Maintained stably
Lip Sync Mismatch in later stages Progressive mismatch Severe mismatch Continuously precise
Accessory/Texture Consistency Appears and disappears intermittently Loss of detail Severe loss Continuously stable

Application Scenarios of SoulX-LiveAct

  • Live Streaming Scenarios: The model can generate digital human streamers in real time, support 24/7 uninterrupted live streaming, synchronize lip movements with speech precisely, and produce natural and rich facial expressions, making it suitable for scenarios such as e-commerce livestreaming, entertainment livestreaming, and knowledge sharing.
  • Virtual Customer Service: The model can provide round-the-clock online service, maintain a stable and consistent digital human appearance, support long-duration conversational interaction, reduce enterprise labor costs, and improve service experience.
  • Podcasts/Talk Shows: Used for producing two-person conversations and interview programs, it generates natural facial expressions and body language in real time, allows guest appearances to be controllable and editable, and quickly produces high-quality content.
  • FaceTime/Video Calls: Can be used in B2B scenarios such as virtual social interaction, online education, and remote meetings, with latency as low as 0.94 seconds and smooth, natural interaction.