Hyper-SD is a framework for accelerating the inference of Diffusion Models (DMs), with the full name Trajectory Segmented Consistency Model for Efficient Image Synthesis. Its goal is to alleviate the computational overhead caused by multi-step inference of diffusion models while maintaining generation quality as much as possible.
Purpose: Hyper-SD is mainly used to accelerate image synthesis based on diffusion models, achieving high-quality image generation under low-step (1 to 8 steps) inference. It is suitable for SDXL and SD1.5 architectures.
Main capabilities:
- Proposes Trajectory Segmented Consistency Distillation, which progressively performs consistency distillation within predefined time-step segments, preserving the original ODE trajectory from a higher-order perspective.
- Introduces human feedback learning to improve the model's performance under low-step settings and mitigate the performance loss caused by the distillation process.
- Integrates score distillation to further enhance low-step generation capability, and for the first time attempts to use a unified LoRA to support inference processes at all step counts.
- Adopts two-stage progressive consistency distillation: in the first stage, consistency distillation is performed separately in the two time intervals [0, T/2] and [T/2, T] to obtain two-segment consistency ODE; in the second stage, this ODE trajectory is used to train a global consistency model.
- On SDXL and SD1.5, 1 to 8-step inference all achieve SOTA performance. For example, Hyper-SDXL at 1-step inference exceeds SDXL-Lightning by +0.68 in CLIP Score and by +0.51 in Aes Score.
- Provides a Real-Time Generation Demo.
- Releases a unified LoRA, compatible with ControlNet, with example conditions including scribble and canny images.
- Qualitative comparisons show that Hyper-SD outperforms other LoRA-based acceleration methods on both SDXL and SD15 architectures and gains more user preference.
Usage: The materials do not provide specific installation commands or code examples, and only mention that a real-time generation demo is provided. Users can visit the official website https://hyper-sd.github.io/ to view the demo.
Applicable scenarios: Suitable for scenarios that require rapid generation of high-quality images, especially inference acceleration of diffusion models based on SDXL or SD1.5. It can be combined with ControlNet for conditional generation, supporting condition inputs such as scribble or canny.
Limitations: The materials do not explicitly list limitations. The performance data are based on specific experiments and user studies, and actual results may vary depending on the base model and configuration.
