AngelSpec - An Open-Source End-to-End Speculative Decoding Training Framework from the Tencent Hunyuan Team

What is AngelSpec

AngelSpec is an open-source end-to-end speculative decoding training framework from the Tencent Hunyuan team, built on TorchSpec, supporting a total of 6 draft architectures: MTP autoregressive and DFly block-parallel. Inference and training are decoupled, and the inference engine streams hidden states to training workers via Mooncake/RDMA. On Hy3-A21B, DFly achieves 1.98–2.40x end-to-end speedup, with throughput improved by 10.5–11.8% over DFlash.

Main Features of AngelSpec

• Unified training for six draft architectures: DFly, DFlash, DFlare, Eagle3, DSpark, and MTP share a single training pipeline, and switching only requires changing the configuration.
• MTP training and TTT unrolling: Supports multi-depth autoregressive unrolled training, with memory close to a single causal forward pass, and supports contexts up to 128k through Ulysses sequence parallelism.
• Acceptance-rate-aligned objective functions: Supports CE, top-k KL, LK Loss, D-PACE weighting, and end-to-end TV loss, which can be freely combined through configuration.
• Document-aware sequence packing: Megatron-style fixed-length packing with strict cross-document isolation, covering both the DFlash and MTP paths.
• Online real evaluation: During training, real speculative decoding is executed through engines such as vLLM, reporting the average acceptance length and per-position acceptance rate in real time.
• Multi-inference backend support: Compatible with mainstream inference engines such as vLLM, SGLang, and HuggingFace.

Technical Principles of AngelSpec

• Speculative decoding fundamentals: AngelSpec is built on the speculative decoding mechanism: a lightweight draft model proposes multiple future tokens in parallel, and the target large model completes batch verification in a single forward pass using rejection sampling, extending single-step decoding into multi-step advancement without changing the target distribution. Its core challenge lies in the trade-off between draft acceptance rate and verification overhead, and AngelSpec systematically addresses this issue from three levels: training, architecture, and inference.
• Workload heterogeneity modeling: AngelSpec takes real-world workload heterogeneity as its primary design consideration. Online requests span open-ended dialogue, code generation, mathematical reasoning, and tool calling, whose conditional entropy and continuation structures differ significantly: in high-entropy dialogue scenarios, the acceptance rate decays rapidly with depth, making short-sequence autoregressive drafts suitable; code and math scenarios contain long predictable spans, making block-parallel diffusion drafts suitable. Therefore, AngelSpec trains complementary specialized drafters—MTP for dialogue data and DFly for code and math data—rather than pursuing a single general-purpose model.
• Shared-parameter multi-depth MTP training: The MTP drafter adopts a shared-parameter multi-depth training scheme. All logical prediction depths reuse the same physical MTP module, and during training it is unrolled autoregressively: depth k+1 receives the argmax prediction from depth k, while maintaining a layer-by-layer growing draft KV cache and a diagonal attention mask. The Training-Time Test (TTT) mechanism exposes the drafter to the self-generated trajectory distribution encountered at inference time, eliminating the exposure bias caused by teacher forcing and significantly improving the acceptance rate of the second and third draft positions.

Follow WeChat and reply “open source” to join theAI Open Source Project Communication Group

How to use AngelSpec

• Environment installation: Execute pip install -e ".[vllm]" and pip install mooncake-transfer-engine to install the framework and the vLLM backend.
• Single-node quick start: Divide 8 GPUs into 4 for inference + 4 for training, run ./examples/qwen3-8b-dfly/run.sh to launch the full pipeline.
• Config override: Override YAML config items directly via CLI, such as training.learning_rate=5e-5 training.num_train_steps=500。
• Conda environment setup: Run ./tools/build_conda.sh 1 vllm to build an isolated environment with mooncake in one click, ready to use after activation.
• Multi-node deployment: Through the Inference Controller and Training Controller scheduled via Ray, combined with Mooncake hidden state storage to achieve cross-node decoupled training.

AngelSpec's core advantages

• Workload specialization: MTP is optimized for high-entropy dialogue scenarios, while DFly is strengthened for scenarios with long predictable spans such as code and math, avoiding a single draft model performing mediocrely across all domains.
• DFly architecture innovation: A hybrid target-conditioned backbone combined with per-layer target views, paired with a predecessor-conditioned autoregressive head, improves intra-block dependency modeling while retaining parallel generation capability.
• D-Cut dynamic verification: Treats target verification as a shared batch-level resource, adaptively adjusting verification depth based on prefix confidence and a runtime cost model, continuously converting extra load into throughput under high concurrency.
• Disaggregated independent scaling: The inference and training GPU pools are decoupled via Mooncake storage, allowing both sides to scale independently according to their own loads, avoiding the suboptimality of a unified parallel strategy.
• Complete open-source suite: Simultaneously releases the framework code, the MTP and DFly draft weights for Hy3-A21B/Qwen3-8B, training configurations, and the technical report, achieving out-of-the-box usability.

AngelSpec's project address

• Project website：https://angelspec.readthedocs.io/
• GitHub repository：https://github.com/Tencent/AngelSpec
• HuggingFace model hub：https://huggingface.co/collections/AngelSlim/angelspec
• arXiv technical paper：https://arxiv.org/pdf/2607.25852

Comparison of AngelSpec with similar competing products

Application scenarios of AngelSpec

• Online inference acceleration for large models: Deploy speculative decoding drafts for MoE/dense models such as Hunyuan Hy3 and Qwen3 to reduce autoregressive decoding latency.
• High-concurrency dialogue services: Leverage D-Cut dynamic resource allocation to maintain the highest average throughput within a concurrency range of 4–64, suitable for customer service and assistant scenarios.
• Code generation and mathematical reasoning: Capture long predictable spans through DFly block-parallel drafting to accelerate structured output tasks such as IDE completion and problem-solving engines.
• Long-context training and inference: Support 128k context with Ulysses sequence parallelism, suitable for long-sequence scenarios such as document analysis and codebase retrieval.