SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

FastMTP - Tencent's Open-Source Large Language Model Inference Acceleration Technology

Author: AI小集 · Original source: AI工具集

What is FastMTP

FastMTP is an in-house large language model (LLM) inference acceleration technology developed by Tencent. By optimizing multi-token prediction (MTP) technology, it replaces traditional multiple independent modules with a single MTP head with shared weights, combined with language-aware vocabulary compression and self-distillation training, significantly improving LLM inference speed by an average of 2.03x with no loss in output quality. FastMTP does not change the main model architecture, is easy to integrate into existing frameworks, applies to structured tasks such as mathematical reasoning and code generation, and provides a practical solution for efficient LLM deployment.

FastMTP

Main Features of FastMTP

  • Significantly Accelerates LLM Inference: By optimizing multi-token prediction (MTP) technology, FastMTP can increase the inference speed of large language models (LLMs) by an average of 2.03x without sacrificing output quality, greatly shortening the time for model content generation and improving application response speed.
  • Maintains Lossless Output Quality: While accelerating inference, FastMTP ensures that the quality of model output is fully consistent with traditional autoregressive generation, without reducing the accuracy or logicality of generated content due to acceleration.
  • Easy to Integrate and Deploy: FastMTP does not require changing the main model architecture; it only needs fine-tuning of a small module and can seamlessly integrate with existing LLM inference frameworks (such as SGLang), greatly reducing deployment cost and difficulty and facilitating rapid application to real-world scenarios.
  • Reduces Hardware Resource Consumption: Based on a single MTP head with shared weights replacing multiple independent modules, it significantly reduces memory usage. At the same time, through language-aware vocabulary compression, it further reduces computation, enabling LLMs to run more efficiently on consumer-grade GPUs and lowering hardware resource requirements.

Technical Principles of FastMTP

  • Speculative Decoding: Drawing on the "draft + verify" strategy, a fast draft model generates multiple candidate tokens, which are then batch-verified by the main model, enabling parallel processing and improving inference efficiency.
  • Single MTP Head with Shared Weights: Abandoning the traditional multi-independent-module design of MTP, it instead uses an MTP head with shared weights to recursively generate multiple tokens, reducing memory usage, forcing the model to learn longer-distance dependencies, and improving draft quality.
  • Self-Distillation Training: Uses data generated by the main model to train the MTP head. Through an exponentially decaying weighted cross-entropy loss function, the MTP head prioritizes learning to generate drafts consistent with the main model's style and logic, improving draft acceptance rate.
  • Language-Aware Vocabulary Compression: During the draft generation stage, it determines the language based on the input context and computes logits only for high-frequency vocabulary, reducing computation. In the verification stage, it uses the full vocabulary to ensure output quality is unaffected.

FastMTP Project Address

  • GitHub Repository：https://github.com/Tencent-BAC/FastMTP
  • HuggingFace Model Library：https://huggingface.co/TencentBAC/FastMTP
  • Technical Paper：https://github.com/Tencent-BAC/FastMTP/blob/main/FastMTP_technical_report.pdf

Application Scenarios of FastMTP

  • Mathematical Reasoning: In mathematical problem-solving scenarios, it rapidly generates solution steps, significantly shortening the time from question input to answer output and improving the response speed of mathematical assistance tools.
  • Code Generation: For programming assistance applications, FastMTP can quickly generate code snippets, helping developers complete code writing faster and improving development efficiency.
  • Long Text Summarization: When handling long-text summarization tasks such as news and articles, quickly extract key information and generate high-quality summaries, saving users' reading time.
  • Multi-turn dialogue: In intelligent customer service or chatbots, accelerate dialogue generation, achieve second-level response, and improve user experience and interaction fluency.