TokenSwift is a lossless acceleration framework for ultra-long sequence generation and has been accepted by ICML 2025. Its core goal is to significantly accelerate the generation process for sequences up to 100K tokens without sacrificing the inherent quality of the target model, reducing computation time from hours to minutes.
Main capabilities:
- Lossless acceleration of ultra-long sequence generation, supporting up to about 100K tokens.
- Reduces the computation time of ultra-long sequence generation from hours to minutes.
- Provides benchmark scripts for sequence lengths of 20K/40K/60K/80K/100K.
- Supports 100% reproducibility for all experiments.
- Provides a pretrained model adapter suitable for arbitrary architectures.
- Releases fine-tuned models: QwQ-32B and DeepSeek-R1-Distill-Qwen-32B, both achieving about 3x speedup, with deployment inference guidelines.
How it works: The framework first lets the target model (LLM), with a partial KV cache and three linear layers, output 4 logits in a single forward pass. It then applies a tree-based attention mechanism to construct candidate tokens. Next, it retrieves top-k candidate 4-grams, which form draft tokens and are fed into the LLM with a full KV cache to generate target tokens. The verification stage is performed by checking whether the draft tokens exactly match the target tokens. Finally, one of the longest valid draft tokens is randomly selected, and the n-gram table and KV cache are updated accordingly.
Usage: Installation provides two methods: Method 1 is installation via pip, and Method 2 is installation from source (recommended). The repository gives getting-started instructions using LLaMA3.1-8B as an example; for other models, scripts in the infer_scripts/ folder can be run. The training guide is optional: filter out data larger than 8K from the PG-19 training set according to different tokenizers, or download the processed training datasets (llama2-pg19, llama3.1-pg19, qwen2.5-pg19); for other models, scripts in the train/scripts/ folder can be run.
Applicable scenarios: Suitable for LLM inference tasks that need to generate ultra-long sequences (up to about 100K tokens), with particular focus on reducing computation time while maintaining generation quality.
Limitations: The materials do not provide specific performance data, hardware requirements, a complete list of supported models, license information, or pricing and charging information.supported models, license information, or pricing and charging information.
