LoRA is a code repository that implements "Low-Rank Adaptation of Large Language Models," containing the source code of the Python package loralib, as well as several examples of integrating it into PyTorch models (such as models in Hugging Face). Currently, only PyTorch is supported.
Purpose LoRA reduces the number of trainable parameters by freezing the original weights and learning a pair of low-rank decomposition matrices. This can significantly lower the storage requirements when adapting large language models to specific tasks, and supports efficient task switching during deployment without introducing inference latency. According to the repository description, LoRA outperforms other adaptation methods such as adapter, prefix-tuning, and fine-tuning.
Main capabilities and results On the GLUE benchmark, when using RoBERTa base/large and DeBERTa XXL 1.5B, LoRA can achieve results comparable to or better than full fine-tuning while training and storing only a small subset of parameters; the repository provides downloads of LoRA checkpoints for RoBERTa and DeBERTa, and notes that the original pretrained checkpoints still need to be obtained from Hugging Face. On GPT-2, LoRA performs well compared with full fine-tuning and efficient tuning methods such as adapter and prefix tuning, with evaluations covering the E2E NLG Challenge, DART, and WebNLG, and it provides downloads of LoRA checkpoints for GPT-2 Medium/Large.
Usage After installing loralib, you can choose to replace some layers with the corresponding implementations in loralib for adaptation; currently only nn.Linear, nn.Embedding, and nn.Conv2d are supported, and MergedLinear is additionally provided for cases where a single nn.Linear represents multiple layers (such as certain attention qkv projection implementations). Before the training loop starts, mark only the LoRA parameters as trainable; when saving a checkpoint, generate a state_dict containing only the LoRA parameters; when loading a checkpoint with load_state_dict, strict=False should be set. The repository also explains that biases can be marked as trainable by passing "all" or "lora_only" to mark_only_lora_as_trainable, and when saving a checkpoint, pass the corresponding bias= parameter to lora_state_dict; LoRA can be applied to any subset of pretrained weights, and the optimal configuration may vary depending on the model architecture and task.
Repository structure loralib/ contains the source code of the package loralib, which needs to be installed before running the examples; examples/NLG/ contains an example implementation of using LoRA in GPT-2, which can be used to reproduce the paper's results; examples/NLU/ contains example implementations of using LoRA in RoBERTa and DeBERTa, which can produce competitive results on the GLUE benchmark.
Other notes Calling model.eval() triggers the merging of the LoRA parameters with the corresponding pretrained weights, thereby eliminating the extra latency of subsequent forward passes; calling model.train() again undoes the merge; passing merge_weights=False to a LoRA layer disables this behavior. A February 2023 update shows that LoRA has gained support from Hugging Face's Parameter-Efficient Fine-Tuning (PEFT) library. Information such as the repository contact and acknowledgments, contribution guidelines (CLA), code of conduct, and security policy can be found in the original repository.
Limitations and content not yet provided Currently only PyTorch is supported. Specific installation commands, complete API details, pricing or charging models, platform support scope, and similar information are not fully provided and are not yet available.
