UltraMem - A New Ultra-Sparse Model Architecture Introduced by ByteDance's Doubao Large Model Team

What is UltraMem

UltraMem is a new ultra-sparse model architecture proposed by theDoubao Large Modelteam at ByteDance, addressing the high memory access issues of traditional MoE architectures during inference. The architecture significantly reduces inference costs by optimizing memory access and computational efficiency, achieving 2-6x faster inference speed than MoE and up to 83% lower costs. The core technologies of UltraMem include: multi-layer structure improvements that split large memory layers into multiple small memory layers distributed across Transformer layers, adding skip-layer operations to enable parallel computation; optimized value retrieval methods using Tucker Decomposition Query-Key Retrieval (TDQKR) to improve retrieval accuracy; and Implicit Value Expansion (IVE), which uses the concepts of virtual memory and physical memory to reduce VRAM and deployment costs. Experiments show that UltraMem demonstrates significant performance advantages across different scales of activated parameters, and its scaling capability surpasses MoE as sparse parameters increase.

Main Features of UltraMem

• Reduced Inference Costs: UltraMem significantly reduces memory access requirements during inference by optimizing memory access mechanisms, lowering inference costs by up to 83%.
• Improved Inference Speed: Compared to traditional MoE architectures, UltraMem achieves 2-6x faster inference speed, and at common batch sizes, memory access costs are nearly equivalent to Dense models with the same computational load.
• Optimized Memory Management: UltraMem reduces memory dependency during inference through sparse computation and selective parameter activation strategies, effectively avoiding memory bottlenecks.
• Support for Large-Scale Models: This architecture opens new paths for building billion-scale value or expert models and possesses excellent scaling characteristics.
• Maintained Model Performance: Under the same parameter and activation conditions, UltraMem significantly reduces inference costs while surpassing MoE in model performance.

Technical Principles of UltraMem

• Sparse Computation and Parameter Decoupling: UltraMem decouples computation and parameters through sparse computation. It activates only the parameters most relevant to the current task, rather than activating all experts during inference like traditional MoE, significantly reducing memory access requirements.
• Optimized Memory Access Mechanism: UltraMem introduces large-scale ultra-sparse memory layers, selectively activating a small number of parameters to avoid memory bottlenecks during inference. This mechanism makes UltraMem's memory access costs nearly equivalent to Dense models with the same computational load at common batch sizes.
• Parallel Computing Mechanism: UltraMem optimizes computation graphs and adopts advanced algorithms to enable multiple inference tasks to proceed simultaneously. The parallel computing mechanism improves resource utilization and further accelerates inference speed.
• Tucker Decomposition Query-Key Retrieval (TDQKR): UltraMem employs a more complex multiplication method—Tucker Decomposition Query-Key Retrieval (TDQKR)—to optimize the value retrieval process. This method improves retrieval accuracy and efficiency by decomposing the interaction between queries and keys.
• Implicit Value Expansion (IVE): UltraMem proposes Implicit Value Expansion (IVE) technology, which implicitly expands sparse parameters through the concepts of virtual memory and physical memory. It enhances model performance and scalability without increasing VRAM burden.
• Multi-Layer Structure Design: UltraMem splits large memory layers into multiple small memory layers, distributing them at fixed intervals across Transformer layers. This allows the model to execute memory layer memory access operations and Transformer layer computation in parallel, improving overall efficiency.

UltraMem Project Address

• arXiv Technical Paper：https://arxiv.org/pdf/2411.12364

UltraMem Application Scenarios

• Real-time Inference Scenarios: UltraMem is suitable for inference scenarios with high latency requirements, such as code completion and intelligent customer service. In these scenarios, the model needs to respond quickly to user requests, and UltraMem's low-latency characteristics can significantly improve user experience.
• Large-scale Model Deployment: UltraMem's efficient inference capability and low memory access cost make it an ideal choice for deploying large-scale language models (LLMs). When building billion-scale value or expert models, UltraMem can achieve high performance under limited computational resources.
• Financial Sector: In the financial sector, UltraMem can be used for credit asset management and risk prediction. By more accurately identifying customer intent and risk characteristics, UltraMem can help financial institutions reduce costs and increase efficiency in their operations.
• Energy Industry: In the energy sector, UltraMem can be applied to equipment inspection, power marketing customer service, and renewable energy power forecasting. Through efficient data processing and inference capabilities, UltraMem can optimize resource allocation and improve energy efficiency.
• Industrial Automation: In industrial automation, UltraMem can be used for equipment fault prediction and production process optimization. Its efficient inference capability can quickly analyze large amounts of industrial data, enabling intelligent production management.