PowerInfer is a high-speed large language model (LLM) inference engine designed for local deployment of LLMs on personal computers equipped with a single consumer-grade GPU. Its core idea is to exploit the high locality inherent in LLM inference: neuron activation follows a power-law distribution, meaning a small number of "hot neurons" are activated for almost all inputs, while the majority of "cold neurons" vary with the specific input. Based on this insight, PowerInfer builds a GPU-CPU hybrid inference engine—hot neurons are preloaded onto the GPU for fast access, while cold neurons are computed on the CPU, thereby greatly reducing VRAM requirements and CPU-GPU data transfer. The engine also integrates an adaptive predictor and neuron-aware sparse operators to further optimize the efficiency of neuron activation and computational sparsity.
Main capabilities and performance:
- Locality-centric design: Leverages sparse activation and the "hot/cold" neuron concept to achieve high-speed inference with lower resource requirements.
- Hybrid CPU/GPU utilization: Seamlessly integrates the memory and compute capabilities of the CPU and GPU to achieve load balancing and faster processing.
- Official evaluation: Running various LLMs (including OPT-175B) on a single NVIDIA RTX 4090 (24G), the average token generation rate is 13.20 tokens/s, with a peak of 29.08 tokens/s, only 18% lower than a top-tier server-grade A100 GPU, and up to 11.69x faster than llama.cpp, while maintaining model accuracy. When running INT4 ReLU models on an RTX 2080Ti (11G), Falcon 40B is up to 8x faster, and Llama 2 70B is up to 3x faster.
Usage and compatibility:
- Easy integration: Compatible with popular ReLU sparse models.
- Convenient local deployment: Designed and deeply optimized for local deployment on consumer-grade hardware, enabling low-latency LLM inference and serving on a single GPU.
- Backward compatibility: Although different from llama.cpp, most examples/ samples (such as server and batch generation) can be used just like llama.cpp. PowerInfer also supports inference using llama.cpp's model weights for compatibility, but without performance gains.
- Model format: PowerInfer models are stored in a special GGUF-based format, PowerInfer GGUFstorage, containing LLM weights and predictor weights. The *.powerinfer.gguf weights and model activation statistics for hot neuron offloading can be obtained from Hugging Face.
- Supported models: ProSparse Llama2 series, ReLU sparse models (such as Falcon(ReLU)-40B-FP16, Bamboo-7B, etc.).
- Supported platforms: x86-64 CPU (with AVX2 instructions) on Linux or Windows, with or without an NVIDIA GPU; Apple M chips (CPU only) on macOS (officially stated as not optimized for Mac, and the performance improvement is currently not significant).
- Build method: use CMake (3.17+), and choose the build command for NVIDIA GPU, AMD GPU (ROCm), or CPU-only according to the hardware.
- Run method: supports CPU-only and CPU-GPU hybrid inference; GPU memory usage can be limited via --vram-budget (replacing llama.cpp's -ngl parameter); supports INT4 (Q4_0) quantized models; supports examples such as serving, batch generation, and perplexity evaluation.
Applicable scenarios: Suitable for developers and researchers who want to run large language models locally on personal computers or consumer-grade hardware and pursue low-latency inference and serving, especially for scenarios with limited VRAM but wanting to run larger models.
Limitations and notes:
- Currently only supports models using ReLU/ReGLU/Squared ReLU activation functions, and does not yet support models such as Mistral, original Llama, and Qwen.
- The current performance metrics of ReLU models (especially 70B) show a clear decline, because fine-tuning used only about 5B tokens (far below the usual training amount of about 2T tokens), and the team is updating better models.
- No optimization has been done on macOS, and the performance improvement is not significant; the Metal backend (for macOS sparse inference) is listed as an upcoming feature.
- If CUDA_ERROR_OUT_OF_MEMORY is encountered, you can try using --reset-gpu-index to rebuild the GPU index, or use a slightly lower --vram-budget value, or use --disable-gpu-index to disable FFN offloading.
- Generated by convert-dense.py, the dense GGUF model may not work properly in llama.cpp because the activation function or model architecture has been modified.
- Officially marked as experimental and no guarantee is provided.
Other information: The project has also released derivative results such as the PowerInfer-2 inference framework for smartphones, the TurboSparse sparse model, and Tiiny AI Pocket Lab. More technical details can be found in its paper.
