What is Miras
Miras is a general framework introduced by Google for deep learning architecture design, especially for sequence modeling tasks. Miras is based on the concepts of associative memory and attentional bias, redefining models such as Transformer and modern linear RNNs as associative memory modules with internal optimization objectives. Miras builds models based on four key choices: associative memory architecture, attentional bias objective, retention gate, and memory learning algorithm. Miras can generate new sequence models with different advantages, such as Moneta, Yaad, and Memora; the models perform excellently in tasks such as language modeling and commonsense reasoning, surpassing existing Transformer and linear RNN models.
Main features of Miras
- Unify existing architectures: Incorporate various existing sequence models (such as Transformer, RetNet, Mamba, etc.) into a unified framework.
- Optimize memory management: Based on introducing the concepts of Attentional Bias and Retention Gate, Miras can better balance learning new information and retaining old information, thereby optimizing the model's memory management capability.
- Design new models: Support the design of new sequence models with different attentional biases and retention mechanisms, such as Moneta, Yaad, and Memora.
- Improve model performance: Improve model performance on long-sequence tasks while maintaining fast parallelized training capability.
Technical principles of Miras
- Associative memory: A mechanism that maps inputs (Keys) to outputs (Values). In Miras, sequence models are viewed as associative memory modules that store and retrieve information based on learning the mapping relationship between inputs and outputs. Associative memory is the core of Miras and determines how the model stores and utilizes information in sequence data.
- Attentional bias: Attentional bias is the internal optimization objective of associative memory, used to measure how the model prioritizes certain events or stimuli. It determines how the model learns the mapping relationship between inputs (keys and values). By choosing different attentional bias objectives (such as ℓ2 regression, ℓ1 regression, Huber loss, etc.), the model's sensitivity and robustness to data are adjusted.
- Retention gate: A regularization mechanism that controls how the model retains old information while learning new information. Introduce retention regularization terms (such as ℓ2 regularization, KL divergence, etc.) to balance learning and retention. Prevent the model from excessively forgetting old information and maintain better performance on long-sequence tasks.
- Memory learning algorithm: The memory learning algorithm is used to optimize the objective function of associative memory. Common algorithms include gradient descent, momentum gradient descent, etc. By choosing a suitable optimization algorithm, the model's training efficiency and convergence speed are improved.
Project address of Miras
- arXiv technical paper：https://arxiv.org/pdf/2504.13173
Application scenarios of Miras
- Language modeling: NLP researchers and text generation developers use it to efficiently process long texts and capture long-distance dependencies.
- Commonsense reasoning: AI researchers and intelligent assistant developers improve the understanding and reasoning ability for implicit information.
- Long text processing: Text analysis engineers and information retrieval experts optimize long text processing efficiency and reduce resource consumption.
- Multimodal tasks: Multimodal researchers and multimedia content analysis engineers integrate multiple modalities of information to improve cross-modal reasoning ability.