TypemovieInfer

TypemovieInfer is a unified, production-oriented multi-GPU inference framework designed specifically for consumer-grade GPUs (RTX 4090/5090), providing optimized multi-GPU parallelism for current advanced image and video generation models.

Purpose

The framework is designed to enable consumer-grade GPUs to efficiently run large image and video generation models, lowering the hardware barrier through multi-GPU parallelism, quantization, and memory optimization, while providing a unified and concise calling interface.

Main Capabilities

• Multi-GPU Acceleration: Supports multiple parallel strategies. Para-Attention (ParaAttn) splits attention computation across multiple GPUs, suitable for FLUX, Kontext, and WAN models; Distributed Data Parallel (DDP) is used for batch distribution of the Qwen model; sequence parallelism (USP) is supported through xfuser integration.
• Memory Optimization: FP8 quantization can reduce memory usage by about 50% with minimal quality loss; intelligent CPU offloading supports 3 levels of granularity; LoRA cache management can efficiently load/unload weights.
• Unified API: All models use a consistent interface, requiring only 3 lines of code to run any model.
• Flexible Configuration: A YAML-based configuration system that supports hot reloading, allowing parameter tuning without modifying code.
• Extensible Architecture: A plugin-based system that allows new models to be added quickly.
• Production Ready: 5/7 models have been tested (71% coverage), providing 4090/5090 performance benchmarks and error diagnostics.

Supported Models

Image Generation

• flux_paraattn: Text-to-image, 2× RTX 4090/5090, default 1024×1024, supports multiple LoRA styles (Film, Realistic, Cartoon, Anime, Sketch, Pixar, Ghibli, etc.), enables FP8 quantization, and automatically prepends trigger words.
• kontext_paraattn: Context-aware image editing, supports text-to-image and image-to-image, can automatically concatenate multiple input images, and supports multiple aspect ratios (16:9, 9:16, 1:1, etc.).
• qwen_ddp / qwen_edit_ddp: Chinese-capable text-to-image, uses multiple GPUs through DDP, provides an image editing variant, and supports multiple aspect ratios. Note: requires QwenImagePipeline in diffusers .

Video Generation

• video_wan_lora_4090 / video_wan_lora_5090: Image-to-video (I2V), 8× RTX 4090 or 8× RTX 5090, generates 81 frames @ 15-16 fps. Supports multiple LoRA categories: basic motion (0_x), talking animation (1_x), pet actions (2_x), landscape animation (3_x), special effects (5x, 6x). The 4090 variant provides an accelerated LoRA option (8 steps vs 30 steps); the 5090 variant enhances stability and improves inference speed by 27%. Enables FP8 quantization and supports CPU offloading memory optimization.

Usage

Environment Requirements

• Python 3.8 or higher
• CUDA 11.8 or higher
• GPU: RTX 4090/5090 or similar (24GB+ VRAM recommended)
• Operating System: Linux (tested), Windows (should work)

Installation

Basic installation and optional feature installation are provided, and verification is required after installation.

Quick Start

Generate an image in 3 lines of code:

model = load_model("flux_paraattn")
images = model.inference(prompt="A beautiful landscape", style_idx=0)

Examples

• Image Generation (FLUX):model = load_model("flux_paraattn"), call inference to pass in prompt and style_idx.
• Image Editing (Kontext):model = load_model("kontext_paraattn"), call inference Pass in prompt and image.
• Video generation (WAN Video):model = load_model("video_wan_lora_5090"), call inference pass in input_image, lora_id, seed.
• Chinese image generation (Qwen):model = load_model("qwen_ddp"), call inference pass in prompt and ratio.

Configuration

Each model has an independent config.yaml, located at typemovie_infer/models/<model_name>/config.yaml. Key parameters include: gpu_num, quant_type, model_path, world_size, num_frames, fps, high_cpu_memory, parameters_level, keep_lora_gpu, quant_model.

Environment variables: MASTER_ADDR (default '127.0.0.1'), MASTER_PORT (default '23456'), NCCL_DEBUG, NCCL_IB_DISABLE. Note that all handlers use the same port 23456 by default; when running multiple models concurrently, a different MASTER_PORT must be set for each model.

Use Cases

• Running large image/video generation models on consumer-grade GPUs.
• Inference tasks that require multi-GPU parallel acceleration.
• Development scenarios that require a unified API to quickly switch between different models.
• Rapid iteration environments that require flexible configuration and hot reloading.

Limitations

• Mainly optimized for RTX 4090/5090; compatibility with other GPUs is not clarified.
• Video generation requires 8 GPUs, and the hardware requirements are relatively high.
• The Qwen model requires a specific version of diffusers support.
• Tested model coverage is 71% (5/7); some models may not be fully verified.
• Multi-model concurrency requires manually setting different ports.
• Performance data is based on tests on specific hardware; actual performance may vary depending on the environment.

Performance Benchmarks

Tested on 8× RTX 4090 (24GB) and 8× RTX 5090 (32GB), with Intel Xeon / AMD EPYC CPUs. Overall performance of the 5090 is 12% faster than the 4090, and the inference stage is 27% faster. Under the 8-GPU setup, memory usage is about 1.65GB per GPU. The ParaAttn model scales linearly with the number of GPUs.

Extension and Contribution

Support is provided for quickly adding new models by creating a model directory, config.yaml, and implementing and exporting the Handler class in handler.py. The project uses the MIT license, and you are welcome to report issues or submit PRs via GitHub Issues.