What is TokenSwift
TokenSwift is an ultra-long text generation acceleration framework launched by the team at the Beijing Institute for General Artificial Intelligence. It can generate 100,000 tokens of text within 90 minutes, a 3x speedup compared to the nearly 5 hours of traditional autoregressive models, with lossless generation quality. TokenSwift reduces model loading latency, optimizes cache update time, and ensures generation diversity through techniques such as multi-token generation and token reuse, dynamic KV cache updates, and context penalty mechanisms. It supports models of various different scales and architectures, such as 1.5B, 7B, 8B, and 14B MHA and GQA models.
TokenSwift's main features
- Ultra-long text generation acceleration: TokenSwift can significantly shorten the generation time of ultra-long text. For example, generating 100,000 tokens of text with a traditional autoregressive model may take nearly 5 hours, while TokenSwift takes only 90 minutes, a 3x speedup that greatly improves generation efficiency.
- Lossless acceleration: While accelerating, TokenSwift maintains the output quality of the original model, ensuring the quality and diversity of the generated text.
- Supports multiple models: TokenSwift supports models of various different scales and architectures, including 1.5B, 7B, 8B, and 14B models, as well as multi-head attention (MHA) and grouped-query attention (GQA) architectures.
TokenSwift's technical principles
- Multi-token parallel generation and token reuse: TokenSwift draws on methods such as Medusa. By introducing additional linear layers, the model can generate multiple draft tokens simultaneously in a single forward pass. Based on the n-gram frequency information in the generated text, the system automatically retrieves and reuses high-frequency phrases, further reducing the number of times the model needs to be reloaded and improving overall efficiency.
- Dynamic KV cache update strategy: In KV cache management, TokenSwift adopts a dynamic update strategy. During generation, the system retains the initial KV cache while replacing subsequent cache entries in an ordered manner according to token importance.
- Tree-structured multi-candidate token verification: To ensure consistency between the generated results and the target model's predictions, TokenSwift introduces a tree attention mechanism. By constructing a tree structure containing multiple candidate token combinations and adopting parallel verification, it randomly selects the longest valid n-gram as the final output, ensuring the generation process is lossless and diversity is improved.
- Context penalty strategy: To further suppress the problem of repetitive generation, TokenSwift designs a context penalty method. During generation, a penalty is applied to recently generated tokens, making the model more inclined toward diverse output when selecting the next token, effectively reducing repetition.
TokenSwift's project addresses
- Project website：https://bigai-nlco.github.io/TokenSwift/
- Github repository：https://github.com/bigai-nlco/TokenSwift
- HuggingFace model library：https://huggingface.co/TokenSwift
- arXiv technical paper：https://arxiv.org/pdf/2502.18890
TokenSwift's application scenarios
- Content creation and copywriting generation: TokenSwift can generate high-quality text content in a short time. For content creators, advertising agencies, and media organizations, it can greatly improve work efficiency.
- Intelligent customer service and chatbots: In intelligent customer service and chatbots, TokenSwift can quickly generate detailed responses. When complex questions or lengthy explanations need to be handled, it can provide a more efficient and more coherent conversational experience.
- Academic research and paper writing: For academic research and paper writing, TokenSwift can help researchers quickly generate literature reviews, experimental reports, or paper drafts, saving a great deal of time and effort.
- Code generation and programming assistance: In the field of programming, TokenSwift can be used to generate complex code snippets or documentation, helping developers quickly build and optimize code.