What is Titans
Titans is a new neural network architecture introduced by Google that can break through the memory bottleneck of Transformers when processing long sequence data. Titans introduces a neural long-term memory module that simulates the memory mechanism of the human brain and strengthens the ability to remember unexpected events. The Titans architecture includes three variants: MAC (Memory as Context), MAG (Memory as Gate), and MAL (Memory as Layer), which integrate the memory module in different ways. Experiments show that Titans surpasses Transformers and modern linear RNN models on tasks such as language modeling, commonsense reasoning, and time series forecasting, especially demonstrating excellent performance in long-sequence tasks with context windows exceeding 2 million, and has parallel computing capability, improving training efficiency.
Main features of Titans
- Long-sequence data processing: Titans can effectively process long-sequence data with context windows exceeding 2 million. In long-sequence tasks, it maintains high accuracy. For example, in the "needle in a haystack" task, even when the sequence length increases from 2k to 16k, the accuracy remains around 90%.
- Memory management: Based on the neural long-term memory module, it remembers information from a long time ago, which is crucial for tasks requiring long-range dependencies, such as language modeling and commonsense reasoning. Combined with the attention mechanism, it handles short-term memory and focuses on direct dependencies in the current context.
- Task diversity:It performs well on a variety of tasks, including language modeling, commonsense reasoning, time series forecasting, genomics modeling, etc., demonstrating broad applicability.
- Training efficiency: Titans' neural long-term memory module supports parallel computing, significantly improving training efficiency, enabling it to process large-scale data faster. In the inference stage, it quickly retrieves and utilizes long-term memory, improving the model's response speed.
Technical principles of Titans
- Neural Long-Term Memory Module
- Memory encoding: Based on an online meta-model, it learns how to remember and forget specific data at test time. The model encodes past information into the parameters of the neural network, avoiding remembering useless details of the training data.
- Surprise metric: Drawing on the principles of human brain memory, it determines the "surprise level" of an input based on the gradient that measures the input. The larger the gradient, the more unexpected the input, and the easier it is to remember.
- Momentum mechanism: It introduces a momentum mechanism that accumulates short-term surprises to form long-term memory, allowing the model to better handle the flow of information in sequences.
- Forgetting mechanism: Based on the forgetting mechanism, the model can erase old memories that are no longer needed, preventing memory overflow and managing limited memory capacity.
- Architecture design
- MAC (Memory as Context): Long-term memory and persistent memory are used as the context of the current input and fed together into the attention mechanism, allowing the model to consider both historical information and the current context at the same time.
- MAG (Memory as Gate): Gated fusion is performed on the two branches of the memory module and sliding window attention. It combines the advantages of long-term memory and short-term memory, using a gating mechanism to dynamically adjust the flow of information.
- MAL (Memory as Layer): The memory module is treated as an independent layer, compressing historical information before feeding it into the attention mechanism. Based on hierarchical information processing, it improves the model's expressive power.
- Parallelized training: Optimized based on matrix operations (matmuls), supports parallel computation, significantly improving training efficiency.
Titans project address
- arXiv technical paper：https://arxiv.org/pdf/2501.00663v1
Titans application scenarios
- Language modeling and text generation: Generate coherent, high-quality long text, such as articles, stories, etc., maintaining consistency and logicality of content.
- Commonsense reasoning and question answering systems: Understand and reason about complex problems in long contexts, provide accurate answers, suitable for question answering tasks that require background knowledge.
- Time series forecasting: Forecast financial markets, weather changes, traffic flow, etc., capture long-period trends, improve forecasting accuracy.
- Genomics and bioinformatics: Analyze DNA sequences, predict protein structures, process long-sequence data in the biomedical field, and facilitate scientific research discoveries.
- Video and music processing: Understand and generate video content, create music, and maintain coherence and style consistency in long sequences.