360-LLaMA-Factory

Purpose

360-LLaMA-Factory is an open-source project developed by the 360 Zhinao team. Based on the widely used LLM post-training framework LLaMA-Factory, it adds Sequence Parallelism (SP), specifically zigzag-ring-attention, to support long-context training. Its goal is to address the training bottleneck when multimodal LLMs process longer token sequences. The project is a direct replacement for LLaMA-Factory, and when SP is not enabled, its behavior is exactly the same as the original version.

Main Capabilities

• Supports most post-training methods, including SFT and DPO (and their variants).
• Sequence parallelism for Qwen3 is now supported (currently only the default zigzag-ring).
• Sequence parallelism for Qwen-VL models (Qwen-2/2.5-VL) is now supported (currently only the default zigzag-ring).
• DeepSpeed Ulysses has been added and tested, with verified correctness and speed.
• Provides full-parameter post-training length benchmarks: on 8 x A800 with SP, zero3-offload, and gradient checkpointing (no other optimizations), full-parameter SFT can reach a maximum length of 210k (7B) and 128k (72B); full-parameter DPO can reach 84k (7B) and 46k (72B).
• Correctness was verified by comparing intermediate values and loss curves (SP=1 and SP=2), and the SFT and DPO loss curves highly overlap.
• Speed test: at a 28k sequence length for a 7B model, DeepSpeed Ulysses and zigzag ring have almost the same training speed, about 15 minutes for 64 steps.

Usage

• Installation: You can install in a fresh Python environment, or in an existing newer LLaMA-Factory environment install at most two additional dependencies and switch to this codebase.
• The degree of sequence parallelism is controlled by the parameter sequence_parallel_size , which indicates the number of GPUs required to process one training data sequence. The default is 1 (SP not enabled). It needs to be set to an integer greater than 1, usually 2, 4, 8, or 16, and must evenly divide the total number of GPUs.
• The parameter cutoff_len now strictly represents the training sequence length and can be set to the maximum length of the data; the data will be automatically padded and preprocessed to adapt to SP.
• These two parameters can be set in the configuration YAML file or on the command line. Refer to 360-example.sh to view example scripts for SFT and DPO.
• Multi-node training is supported and can be launched using DeepSpeed.

Applicable Scenarios

• Large model research and development that requires post-training (SFT, DPO, etc.) on long-sequence data.
• Long-context training for multimodal large models.
• Those who want to conveniently combine optimization tools such as unsloth or liger-kernel in a parameterized way.

Limitations

• Due to limitations of Ring Flash Attention, under sequence parallelism the parameter attention_dropout is currently not supported; if this parameter is non-zero when sequence parallelism is enabled, it will be reset to zero.
• Currently, SP for Qwen3 and Qwen-VL only supports the default zigzag-ring mode.
• When SP is not enabled, the behavior is consistent with the original LLaMA-Factory.
• Other SP modes (such as the llama3 style) are still planned.
• In loss calculation, zigzag ring attention has numerical instability in the first forward pass, which has been resolved by overriding training_step() and performing one no-effect empty forward pass.
• Some current optimization items (such as precomputing reference model logits, logits.float()/contiguous() optimization, full compatibility with unsloth/liger kernel, SP support for train/pt, rm, and kto, etc.) are still on the to-do list.