Firefly

Firefly is an open-source one-stop large model training tool, aimed at developers and researchers who want to train or fine-tune large language models on their own. It supports pretraining, instruction fine-tuning (SFT), and DPO for mainstream open-source large models, including but not limited to Qwen2, Qwen1.5, Qwen2.5, Yi-1.5, Yi, Llama3, Llama2, Llama, Gemma, MiniCPM, MiniCPM3, Phi-3, InternLM, Baichuan, Baichuan2, ChatGLM, Deepseek, Orion, Ziya, Xverse, Mistral, Mixtral-8x7B, Zephyr, Vicuna, Bloom, etc.

1. Main capabilities

2. Training tasks: Supports pretraining, instruction fine-tuning (SFT), and DPO. Pretraining uses the classic autoregressive loss, where tokens at all positions participate in loss computation; instruction fine-tuning only computes the loss for the assistant response part.

3. Training modes: Supports full-parameter training, LoRA, and QLoRA efficient training. The project recommends using QLoRA for instruction fine-tuning when training resources are limited, and this method has been verified as effective on the Open LLM Leaderboard.

4. Model adaptation: During training, it aligns with the template of each official chat model, and the code is adapted for the templates of different chat models.

5. Unsloth acceleration: Supports using Unsloth to accelerate training and save GPU memory. According to the project description, fine-tuning Llama3-8B with Unsloth requires only 7.75 GB of GPU memory, can reduce memory usage by 42.58%, and reduce training time by 30.72%; on Qwen2, it speeds up by 47.32%, saves 39.13% of GPU memory, and requires at least only 8.43 GB of GPU memory. The PR for the Unsloth x Qwen2 model structure has been merged into Unsloth's master branch and extends Unsloth's support for the Qwen2 model structure, including the Dense models of the Qwen1.5 series.

6. Datasets and model weights: Organizes and open-sources instruction fine-tuning datasets firefly-train-1.1M, moss-003-sft-data, ultrachat, WizardLM_evol_instruct_V2_143k, school_math_0.25M, unified into the same data format; open-sources the Firefly series instruction fine-tuning model weights, such as firefly-mixtral-8x7b (Open LLM Leaderboard score 70.34), firefly-baichuan2-13b, firefly-llama-30b, firefly-llama2-13b, firefly-llama-13b, etc.

7. Related projects: LongQLoRA is used to efficiently extend the LLama context length; Firefly-LLaMA2-Chinese performs Chinese vocabulary expansion, incremental pretraining, and multi-turn instruction fine-tuning on Llama2; LLMPruner is a pruning tool for large language models and performs vocabulary pruning on Bloom.

8. Usage Hmm, the project trains different models through configuration files. The training parameter configurations are stored in the train_args directory and can be modified or added as needed. The main parameters include output_dir, model_name_or_path, train_file, template_name, num_train_epochs, deepspeed, train_mode (full, lora, or qlora, default qlora), task_type (pretrain, sft, or dpo, default sft), per_device_train_batch_size, gradient_accumulation_steps, gradient_checkpointing, learning_rate, max_seq_length, max_prompt_length, logging_steps, save_steps, save_total_limit, lr_scheduler_type, warmup_steps, optim, seed, fp16, bf16, use_unsloth, etc. When training with QLoRA, lora_rank, lora_alpha, and lora_dropout also need to be set. The training data is in jsonl format, and the conversation field is required. You can refer to data/dummy_data.jsonl; for the pretraining data format, refer to data/pretrain/dummy_pretrain.jsonl; for the DPO data format, refer to data/dummy_dpo.jsonl. Run the script under requirements.txt to install dependencies. For most models, debugging and training are done in the torch==1.13 and transformers==4.36 environment; Qwen1.5 requires updating transformers to 4.37; Gemma requires updating transformers to 4.38.1 and torch==2.0.0. When using QLoRA to train Baichuan2, you need to install torch==2.0 and uninstall xformers and apex; when using QLoRA to train Qwen, you need to uninstall flash-attn. If Unsloth is enabled, you need to install or update the relevant Python packages according to the instructions. If LoRA or QLoRA is used for training, the project only saves the adapter weights and configuration files, and you need to use script/merge_lora.py to merge the adapter weights with the base model. The project provides a multi-turn dialogue interaction script, located in the script/chat directory, which is compatible with inference for all models trained in this project; the template_name in the script must be consistent with that used during training, and parameters such as top_p, temperature, repetition_penalty, and do_sample have a large impact on generation quality and can be tuned according to the scenario. The inference script supports inference using the base model and adapter, and also supports inference using 4bit.

III. Applicable Scenarios It is suitable for research and development scenarios that require pretraining, instruction fine-tuning, or DPO on mainstream open-source large models; it is suitable for scenarios with limited VRAM or computing power and where training costs are expected to be reduced through LoRA/QLoRA or Unsloth; it is also suitable for scenarios where experiments are expected to be carried out using open-source Chinese instruction datasets and already open-sourced model weights.

IV. Limitations and Notes The project description lists that, due to factors such as model parameter size limits and the degree of training data cleaning, open-source models may have the following limitations: they are prone to producing incorrect responses to factual knowledge; because they have not undergone harmlessness fine-tuning, they may produce discriminatory, harmful, or unethical statements; and their capabilities in code and reasoning are still lacking. Based on the above limitations, the project requiresthat the code, data, and models must not be used for purposes that cause harm to society, and the commercial license of the base model should be followed. If OOM occurs during training, parameters such as per_device_train_batch_size and max_seq_length can be reduced, or gradient_checkpointing=true can be set. If an error occurs during training, you can first check the FAQ. If you need to specify the use of GPUs 0 and 1 for training, it can be set through the relevant method. When using Qwen-Base and Yi-Base, it is recommended to set template_name="default" to avoid specific issues; SFT on Qwen-Chat and Yi-Chat models will not produce this issue, and they can be set to "qwen" and "yi" respectively. This issue does not exist in Qwen1.5. If you use the data, code, or models of this project, you need to cite this project.