What is AngelSlim
AngelSlim is a full-modality large model compression toolkit independently developed and open-sourced by the Tencent Hunyuan team. Through technologies such as quantization, speculative sampling, sparsification, and distillation, it provides efficient and easy-to-use compression and acceleration solutions for large language models (LLM), vision-language models (VLM), and speech models. The toolkit integrates mainstream compression strategies such as FP8/INT8/INT4 quantization, GPTQ, and AWQ, supports one-click invocation and end-to-end deployment, and the models produced by training can seamlessly connect with inference frameworks such as vLLM and Sglang. AngelSlim has significantly upgraded its speculative sampling training framework, pioneering the Eagle3 architecture and extending speculative sampling technology to full-modality scenarios for the first time, enabling a small model to learn to draft multi-step candidate tokens for a large model, which are then verified in parallel by the large model. In measured tests, inference speed can be increased by up to 1.4-1.9 times. It currently supports compression optimization for mainstream models such as the Hunyuan, DeepSeek, and Qwen series. Developers can quickly install and use it via pip install angelslim .
AngelSlim's main features
-
Multi-precision quantization compression: Supports quantization at multiple precisions such as FP8, INT8, and INT4, and integrates mainstream quantization algorithms such as GPTQ and AWQ, greatly reducing model storage and computation overhead.
-
Speculative sampling acceleration: Pioneers the Eagle3 training architecture, enabling a small model to draft multi-step candidate tokens for a large model, which are verified in parallel by the large model, increasing inference speed by up to 1.4-1.9 times.
-
Full-modality coverage support: Supports compression and acceleration for large language models (LLM), vision-language models (VLM), and speech models (ASR/TTS), realizing for the first time the application of speculative sampling in full-modality scenarios.
-
Sparsification and distillation: Integrates technologies such as structured/unstructured sparsity and knowledge distillation to further compress model size while preserving performance.
-
One-click compression invocation: Provides highly integrated API interfaces, simplifying the invocation process of mainstream compression algorithms and lowering the barrier to use for developers.
-
End-to-end deployment integration: Models produced by training can be directly used with mainstream inference frameworks such as vLLM and Sglang, achieving seamless connection from compression to deployment.
-
Multi-model ecosystem compatibility: Already supports compression optimization for mainstream open-source models such as Hunyuan, DeepSeek, Qwen, Qwen2.5VL, and Qwen3-Omni.
AngelSlim's technical principles
-
Quantization compression principles: By mapping model weights from high-precision floating-point numbers (FP16/FP32) to low-precision integers (INT8/INT4) or the FP8 format, it reduces storage usage and computation, while using algorithms such as GPTQ (layer-wise quantization optimization) and AWQ (activation-aware weight quantization) to minimize precision loss.
-
Speculative Decoding principles: Train a small draft model to generate multiple candidate tokens in advance, and then have the large target model verify them in parallel at once, accepting valid tokens, rejecting incorrect tokens, and rolling back and correcting them, breaking through the serial decoding bottleneck through the collaborative mechanism of "small model drafts + large model checks".
-
Eagle3 architecture principles: Introduces a "look-ahead" training strategy on top of standard speculative sampling, enabling the draft model to learn to predict the distribution characteristics of multiple future steps of tokens and improving the quality of candidate sequences, thereby increasing the acceptance length of the large model (up to 1.8-3.5 times) and achieving a higher speedup ratio.
-
Knowledge distillation principles: Transfer the output distribution and intermediate-layer features of a large model (teacher model) to a small model (student model), and through soft-label supervision and feature alignment, enable the small model to inherit the capabilities of the large model while compressing its size.
-
Sparsification Principles: Remove redundant weight connections through pruning (unstructured sparsity) or entire neurons/attention heads (structured sparsity), reducing the model's computational density and, combined with hardware acceleration, achieving inference speedup.
-
Principles of a Unified Omni-Modal Framework: In response to the architectural differences among models for different modalities (text, images, speech), design a unified compression interface and modality adaptation layer so that techniques such as quantization and speculative sampling can be reused across modalities.
AngelSlim Project Address
-
GitHub Repository：https://github.com/Tencent/AngelSlim
-
Hugging Face Model Hub：https://huggingface.co/collections/AngelSlim
AngelSlim Application Scenarios
-
Cloud-based Large Model Inference Acceleration: Provide quantization compression and speculative sampling acceleration for hundred-billion-parameter large models such as Hunyuan, DeepSeek, and Qwen, reducing GPU memory usage and computational latency and improving service throughput in high-concurrency scenarios.
-
On-Device Model Deployment Optimization: Through INT4/INT8 quantization, compress large models to a size suitable for running on phones, tablets, and IoT devices, supporting offline inference and privacy-preserving scenarios.
-
Multimodal AI Application Acceleration: Provide omni-modal compression solutions for image-text understanding in vision-language models (VLMs) and real-time transcription and synthesis in speech models (ASR/TTS), accelerating applications such as intelligent customer service, content moderation, and real-time translation.
-
AIGC Content Generation Efficiency Improvement: In generative scenarios such as AI writing, code generation, and image captioning, significantly reduce first-token latency and overall generation time through speculative sampling technology, improving user experience.
-
Cost Reduction for Enterprise Private Deployment: Help enterprises compress open-source large models and deploy them to private clouds or local servers, reducing computing procurement costs and achieving cost-effective AI middle-platform construction.