dash-infer

DashInfer is a native large language model (LLM) inference engine, with its runtime written in C++, designed to provide production-grade, highly optimized implementations for various hardware architectures (including CUDA, x86, and ARMv9), with the goal of achieving industry-leading performance across multiple hardware platforms.

Purpose and Positioning DashInfer is used to efficiently deploy and run large language model inference services in production environments, emphasizing a lightweight architecture, high precision, and high performance.

Main Capabilities

• Lightweight architecture: Very few third-party dependencies, with almost all dependent libraries statically linked, providing C++ and Python interfaces, making it easy to integrate into existing systems.
• High precision: Rigorously tested, with inference precision consistent with PyTorch and other GPU engines (such as vLLM).
• High-performance inference technologies:
  • Continuous Batching: Allows new requests to be inserted immediately and supports streaming output.
  • Paged Attention: Uses the self-developed SpanAttention technology, combined with int8 and uint4 KV cache quantization, accelerated based on efficient GEMM and GEMV implementations.
  • Prefix Cache: Supports efficient prefix caching for prompts, and can be used for standard LLMs and multimodal language models (such as Qwen-VL), while utilizing both GPU and CPU.
  • Quantization support: Enables weight-only quantization acceleration through InstantQuant (IQ), without the need for fine-tuning, improving deployment efficiency; precision evaluations show that IQ has almost no impact on model precision.
  • Asynchronous interface: A request-based asynchronous interface that can individually control the generation parameters and request status of each request.
• Supported models:
  • Mainstream open-source LLMs: Qwen, LLaMA, ChatGLM, etc., with support for loading models in HuggingFace format.
  • Multimodal language models (MMLM): Qwen-VL, Qwen-AL, Qwen2-VL, etc.
• OpenAI API Server: Can be combined with fastChat to implement an OpenAI-compatible API server.
• Multilingual API: Provides C++ and Python interfaces, and the C++ interface can be extended to other languages such as Java and Rust through standard cross-language interfaces.

Supported Hardware and Data Types

• CUDA GPU: Supports CUDA 11.4 - 12.4, and supports multiple compute architectures such as SM70 - SM90a (T4, 3090, 4090, V100, A100, A10, L20, H20, H100). Data types: FP16, BF16, FP8, FP32, Int8(InstantQuant), Int4(InstantQuant).
• x86 CPU: Requires the AVX2 instruction set; for Intel 5th Gen Xeon (Emerald Rapids) and 4th Gen Xeon (Sapphire Rapids), corresponding to Alibaba Cloud 8th Gen ECS instances (such as g8i), AMX instructions are used to accelerate computation. Data types: FP32, BF16.
• ARMv9 CPU: Requires the SVE instruction set; supports ARMv9 architecture processors such as Yitian 710, corresponding to Alibaba Cloud 8th Gen ECS instances (such as g8y), and uses SVE instructions to accelerate computation. Data types: FP32, BF16, Int8(InstantQuant).

Quantization Technology DashInfer provides a variety of LLM weight quantization technologies, such as int8/int4 weight-only quantization, int8 activation quantization, and custom fused kernels optimized for specific devices. Supported quantization algorithms include:

• InstantQuant (IQ): A dynamic quantization technology that requires no fine-tuning and enables faster deployment.
• GPTQ: Models fine-tuned with GPTQ have better accuracy, but require a fine-tuning step. Quantization strategies are divided into:
• Weight Only Quantization: Only quantizes and compresses weights, using bf16/fp16 during computation, reducing memory access but not improving computational performance compared with BF16.
• Activation Quantization: Weights are stored as int8, and low-precision quantization computation is performed during the computation stage, which can reduce memory access and improve computational performance, but accuracy may decrease slightly, requiring business data accuracy testing. Quantization granularity includes:
• Per-Channel (Per-Token): DashInfer quantization technology uses at least this granularity, which is simple and has optimal performance, and can usually meet most accuracy requirements.
• Sub-Channel: Divides channels into N groups, with quantization parameters computed for each group. Accuracy is better, but implementation is complex and limited, and performance may be slightly slower; activation quantization is difficult to implement due to formula constraintsSub-Channel (DashInfer's activation quantization is all Per-Channel).

Usage

• Provides documentation and sample code, covering C++ and Python interfaces.
• Python Quick Start, LLM OpenAI Server Guide, VLM OpenAI Server support.
• Feature introduction: Guided Decoding.
• Development guide: Building from source, environment variables.
• Examples are located at <path_to_dashinfer>/examples, and documentation is located at <path_to_dashinfer>/documents/EN.
• Basic GPU Python examples, documentation for all Python examples, C++ example documentation.
• Multimodal model (VLM) support: The multimodal folder provides a VLM inference toolkit based on the DashInfer engine, compatible with the OpenAI Chat Completion API, and supports text and image/video input.

Applicable Scenarios Suitable for production environments that need to deploy high-performance LLM inference services on multiple hardware types (CUDA GPU, x86 CPU, ARMv9 CPU), especially for scenarios with accuracy requirements, scenarios that want to leverage technologies such as continuous batching, prefix caching, and quantization acceleration, and scenarios that require multimodal model inference (such as image/video input).

Limitations The materials do not explicitly list limitations, and only mention that some quantization strategies have performance or implementation limitations (such as Sub-Channel activation quantization being difficult to implement). Specific limitations are not provided for now.

Version Updates

• [2024/12] v2.0 released, enhancing GPU (CUDA) support, including prefix caching (GPU and CPU swapping), guided decoding, optimized attention for GQA, a lock-free reactor engine, and adding support for VLM models (Qwen-VL) and MoE models.
• [2024/06] v1.0 released, supporting x86 and ARMv9 CPUs and CPU flash attention.

License The DashInfer source code uses the Apache 2.0 license.

Upcoming Features Multimodal model support, using Flash-Attention to accelerate attention, expandingExtend the context length to over 32k, support 4-bit quantization, support quantized models fine-tuned with GPTQ, support MoE architecture, guided output (JSON mode), prefix caching (GPU prefix caching and CPU swapping), quantization (FP8 A8W8 activation quantization support on CUDA), LORA (continuous batching LORA optimization), in-engine parallel context phase and generation phase, more effective GPU MoE operators, porting to the AMD (ROCm) platform.

Performance Benchmarks The documentation provides performance comparisons with mainstream LLM inference engines, including comparisons between Qwen-VL and vLLM at different model scales (2B/7B using A100-80Gx1, 72B using A100-80Gx4), as well as the reduction in TTFT (time to first token) under different prefix cache hit rates (model Qwen2-72B-Instruct, input 4000 tokens, output 1 token). It also compares the output of guided decoding (JSON mode) under different engines (context length 45, generation length 63).

Related technologies: HIE-DNN (high-performance DNN inference operator library), SpanAttention (high-performance decoding-phase attention based on paged KV cache), MoE operators (see the paper, using the efficient top-k operator RadiK).