What is MeteoRA
MeteoRA is a multi-task embedding framework for large language models (LLMs) launched by the research team of the Department of Computer Science and Technology at Nanjing University. It integrates multiple task-specific LoRAs (low-rank adapters) into a single base model, achieving efficient parameter reuse and autonomous task switching. MeteoRA is based on a mixture-of-experts (MoE) architecture, using a trainable gating network to dynamically select the LoRA adapter most suitable for the current input, without the need for explicit task instructions. MeteoRA proposes a MoE forward acceleration strategy based on custom GPU kernel operators, significantly improving inference efficiency while maintaining low memory overhead. In experiments, MeteoRA demonstrates performance comparable to traditional fine-tuning methods across a variety of tasks, excels in composite tasks, and supports solving multiple sub-problems within a single inference process.
MeteoRA's main features
- Multi-task adapter integration: Embeds multiple task-specific LoRAs (low-rank adapters) into a single base LLM, handling multiple tasks simultaneously.
- Autonomous task selection and switching: Enables autonomous task switching without manually specifying task intent.
- Efficient inference: Improves the inference efficiency of multi-task adapters while maintaining low memory overhead.
- Composite task processing: Solves multiple sub-tasks in a single inference, such as continuously answering multiple questions from different domains, improving the model's flexibility and practicality.
- Scalability: Supports the integration of multiple LoRA adapters, applicable to different tasks and domains, expanding the application scenarios of LLMs.
MeteoRA's technical principles
- LoRA（Low-Rank Adaptation）: A parameter-efficient fine-tuning method based on injecting low-rank matrices (A and B) into the linear layers of an LLM, updating only the matrices to adapt to specific tasks without changing other parameters of the base model. Each LoRA adapter contains a pair of low-rank matrices used to modify the model's output.
- Mixture-of-experts (MoE) architecture: Based on the MoE architecture, multiple LoRA adapters are treated as different "experts", and a trainable gating network dynamically selects the expert (LoRA adapter) most suitable for the current input. The gating network calculates the weight of each LoRA based on the input hidden state, and selects the top-weight adapters for forward propagation.
- Dynamic gating mechanism: The gating network dynamically assigns weights to each input, determining which LoRA adapters participate in computation. Based on a top-k selection strategy, it supports selecting multiple adapters, enabling flexible task switching and combination.
- Forward acceleration strategy: Introduces custom GPU kernel operators based on PyTorch and Triton. The operators are based on parallelization and optimized memory access, improving the speed of multi-task inference while maintaining low memory usage.
- All-mode integration: Embeds LoRA adapters into all linear layers of the Transformer architecture (including attention modules and MLP modules), making more comprehensive use of knowledge from different tasks.
MeteoRA's project address
- GitHub repository：https://github.com/NJUDeepEngine/meteora
- arXiv technical paper：https://arxiv.org/pdf/2405.13053
MeteoRA's application scenarios
- Multi-domain question answering: Integrates knowledge from different domains, automatically switches adapters, and accurately answers various questions.
- Multilingual dialogue: Supports translation between multiple language pairs, enabling fluent multilingual communication.
- Composite task processing: Solve complex problems containing multiple subtasks, dynamically switching adapters to complete tasks.
- Cross-domain knowledge fusion: Combine knowledge from different domains to enhance complex task processing capabilities.
- Intelligent customer service and assistants: Dynamically switch adapters according to user needs, respond quickly, and improve service quality.