What is MindLLM
MindLLM is an AI model jointly launched by Yale University, Dartmouth College, and the University of Cambridge that can decode brain functional magnetic resonance imaging (fMRI) signals into natural language text. MindLLM is based on a subject-agnostic fMRI encoder and a large language model (LLM) to achieve high-performance decoding, and introduces Brain Instruction Tuning (BIT) technology to capture diverse semantic information in fMRI signals. MindLLM performs excellently on multiple benchmarks, with downstream task performance improved by 12.0%, cross-individual generalization ability improved by 16.4%, and adaptability to new tasks improved by 25.0%. MindLLM provides new possibilities for brain-computer interfaces and neuroscience research.
MindLLM's main features
- Brain activity decoding: Converts the neural activity of the brain during perception, thinking, or recall into intuitive textual descriptions, helping scientists and doctors better understand how the brain works.
- Cross-individual universality: Processes brain signals from different individuals without the need for separate training for each individual, greatly improving the model's generalization ability.
- Multi-functional decoding: MindLLM adapts to multiple tasks, such as visual scene understanding, memory retrieval, language processing, and complex reasoning, demonstrating strong multi-functionality.
- Assisted healthcare and human-computer interaction: Restores communication ability for patients with aphasia, or controls devices such as prosthetics and virtual assistants based on neural signals, advancing the development of brain-computer interface technology.
MindLLM's technical principles
- fMRI encoder: Uses a neuroscience-inspired attention mechanism to encode fMRI signals into a series of 'brain feature tokens'. The encoder learns the functional information and spatial location information of different brain regions, dynamically extracts features, and avoids information loss caused by individual differences.
- Large language model (LLM): Combines the encoded brain feature tokens with a language model, and based on the powerful generation capability of the LLM, converts brain signals into natural language text. A pretrained LLM (such as Vicuna-7b) is used as the decoder to ensure that the generated text has semantic coherence and accuracy.
- Brain Instruction Tuning: Trains the model based on diverse datasets (such as visual question answering, image captioning, and memory retrieval tasks) to capture diverse semantic information in fMRI signals. The BIT dataset uses images as an intermediary to pair fMRI data with corresponding text annotations, training the model to perform multiple tasks and improving its multi-functionality and adaptability.
- Subject-agnostic design: Based on the functional information of separated brain regions (consistent across individuals) and fMRI signal values, MindLLM shares prior knowledge across different individuals, achieving universal cross-individual decoding capability.
MindLLM's project address
- arXiv technical paper：https://arxiv.org/pdf/2502.15786
MindLLM's application scenarios
- Medical rehabilitation: Restores communication ability for patients with aphasia, paralysis, and other conditions, and decodes brain signals to help users express thoughts or control external devices.
- Brain-computer interface: Develops more efficient and more intuitive brain-computer interface systems, such as controlling prosthetics, wheelchairs, or virtual reality devices, improving the quality of life for people with disabilities.
- Neuroscience research: Helps scientists better understand the brain's cognitive mechanisms, conscious activity, and the relationship between neural signals and behavior, advancing the development of neuroscience.
- Human-computer interaction: Realize a more natural and more direct way of human-computer interaction, using brain signals to control electronic devices, smart homes, or autonomous driving systems, enhancing the interaction experience.
- Mental health assistance: Monitor and analyze brain activity, assist in diagnosing mental illnesses or evaluating treatment effects, and provide new tools and methods for the field of mental health.