AngelSpec

1. Project Overview

AngelSpec is Tencent's PyTorch-native training framework for building speculative-decoding draft models — the small "predictor" models that let large language models generate multiple tokens per forward pass instead of one, cutting inference latency without touching output quality.

2. Background & Positioning

Speculative decoding speeds up LLM inference by having a lightweight draft model propose several tokens ahead, which the full target model then verifies in a single pass. The speedup only materializes if the draft model is well aligned with its target — and training that alignment well, at production scale, has historically required stitching together research code. AngelSpec was created by Tencent's Hunyuan AI Infra team to close that gap: it treats speculative decoding as a first-class training objective, not a post-hoc inference trick, and productizes the training pipeline behind Tencent's own Hy3 and Qwen3 draft-model releases.

What sets it apart from other speculative-decoding trainers:

• Architecture breadth in one framework — six distinct draft architectures (autoregressive and block-parallel) are selectable through configuration rather than requiring separate codebases.
• Disaggregated training design — inference (hidden-state generation) and training (optimization) run on separate GPU worker groups connected by a high-throughput tensor store, so each can scale independently.
• Production pedigree — it is the framework used to train and release Tencent's own Hy3-A21B and Qwen3-8B draft models, not a research prototype.

3. Feature Categories

• 🧩 Draft Architectures — 6 selectable methods. Representative examples: DFly (block-parallel with hidden-correction AR head), DFlash (anchor sampling + parallel block generation), DFlare (DFlash with learnable per-layer target fusion), Eagle3 (autoregressive test-time training), DSpark (DFlash backbone with an EAGLE-style AR head), MTP (full MoE decoder layer as the draft head). Purpose: match the draft method to the target model and latency budget without switching frameworks.
• ⚙️ Training Configs — 5+ ready-to-run YAML configs. Examples: vllm_qwen3_8b_dfly.yaml, vllm_qwen3_8b_dflare.yaml, sglang_qwen3_8b_dflash.yaml, sglang_qwen3_8b_dspark.yaml, vllm_qwen3_8b_mtp_pack_usp_40k.yaml. Purpose: reproduce known-good training recipes for common target models and backends.
• 📦 Runnable Examples — 2 target-model example suites (Hy3 multi-node, Qwen3-8B single-node) covering DFly, DSpark, MTP, and DFly-CPT variants. Purpose: give newcomers a working end-to-end training run to start from.
• 🤗 Released Draft Models — multiple pre-trained checkpoints on Hugging Face, including Hy3-DFly-Block8 and Hy3-MTP-TTT3, in both standard and "think" modes. Purpose: let users evaluate or deploy speculative decoding immediately, without training from scratch.

4. Key Highlights

• Long-context MTP training with TTT — multi-token-prediction training uses on-policy, multi-depth rollout with a memory-efficient single causal pass, scaling to 128k-token contexts via Ulysses sequence parallelism.
• Acceptance-aligned loss functions — composable losses (cross-entropy, top-k KL, LK losses, D-PACE weighting) are tuned to directly optimize the metric that matters for speculative decoding: how many draft tokens the target model actually accepts.
• Document-aware sequence packing — Megatron-style fixed-length packing with cross-document isolation keeps training throughput high without leaking attention across unrelated documents.
• Online speculative-decoding evaluation — training runs report genuine speculative decoding metrics (e.g., mean accepted length) during training, not just proxy losses, so drafter quality is visible in real time.
• Disaggregated inference/training via Mooncake — separate GPU worker groups for hidden-state generation and model optimization communicate through a Mooncake tensor store, allowing each side to scale independently on large clusters.
• Measured end-to-end speedups — DFly delivers up to 2.40× average end-to-end speedup over autoregressive decoding, with benchmarked throughput gains that hold up under real traffic and at high concurrency.

5. Use Cases by Role

• General developers — integrate a released draft model (e.g., a Qwen3-8B variant) with vLLM or SGLang to speed up existing inference pipelines without retraining anything.
• DevOps / SRE — use the disaggregated worker-group design to plan GPU cluster capacity separately for hidden-state generation and training, and monitor online acceptance-length metrics as a production health signal.
• Data / research scientists — train custom draft models against proprietary target models, experiment across the six supported architectures, and tune acceptance-aligned loss weighting for a specific workload.
• Project managers — evaluate AngelSpec's published benchmarks and released checkpoints to scope inference-cost reduction initiatives before committing engineering time to custom training.

6. Getting Started

Find what you need — browse the repository's configs and examples, or check the released checkpoints on Hugging Face:

https://huggingface.co/collections/AngelSlim/angelspec

Install / integrate:

pip install -e ".[vllm]"
pip install mooncake-transfer-engine
./examples/qwen3-8b-dfly/run.sh

Contribute — fork the repository, make changes, and open a pull request:

git clone https://github.com/Tencent/AngelSpec

7. Project Structure

AngelSpec/
├── angelspec/     # Core framework code (architectures, losses, training loop)
├── configs/       # Training configuration YAML files
├── examples/      # Runnable end-to-end examples (Hy3, Qwen3-8B targets)
├── tests/         # Test suite
├── docs/          # Documentation source
└── tools/         # Build and utility scripts (e.g. build_conda.sh)

8. Related Ecosystem

• TorchSpec — the PyTorch-native foundation AngelSpec builds on for speculative-decoding training primitives.
• Mooncake — provides the tensor-store transfer engine that connects AngelSpec's disaggregated inference and training GPU worker groups.
• vLLM / SGLang — supported inference backends for both training-time rollout and deploying the resulting draft models.
• AngelSlim — Tencent's broader large-model compression toolkit, of which AngelSpec is a part.

9. License

• ✅ Use, modify, and distribute AngelSpec's own code, including for commercial purposes, under the Apache-2.0 license.
• ✅ Build and train your own draft models on top of the framework.
• ❌ Do not remove existing copyright or license notices when redistributing.
• ℹ️ Bundled third-party components carry their original licenses (Apache-2.0 for Eagle and Transformers; MIT for SpecForge, DeepSpec, and TorchSpec) — check the relevant subdirectory before redistributing those parts separately.

10. FAQ

Q: Which draft architecture should I start with?
A: For most target models, DFly is a reasonable default — it delivers the largest published speedup (up to 2.40×) and has ready-made configs like vllm_qwen3_8b_dfly.yaml.

Q: Do I need to train my own draft model to use AngelSpec?
A: No. Pre-trained checkpoints (e.g., Hy3-DFly-Block8, Hy3-MTP-TTT3) are published on the Hugging Face collection and can be deployed directly.

Q: What inference backends does AngelSpec support?
A: Both vLLM and SGLang, selectable per config (e.g., vllm_qwen3_8b_dfly.yaml vs. sglang_qwen3_8b_dflash.yaml).

Q: How does AngelSpec handle very long training contexts?
A: MTP training with TTT scales to 128k-token contexts using Ulysses sequence parallelism combined with a memory-efficient single causal pass.

Q: Where can I read the technical details behind the design?
A: The accompanying paper, "AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding," is available at arXiv:2607.25852.

11. Quick Links

• Repository: https://github.com/Tencent/AngelSpec
• Documentation: https://angelspec.readthedocs.io
• Technical report: https://arxiv.org/abs/2607.25852
• Pre-trained models: https://huggingface.co/collections/AngelSlim/angelspec
• Issues: https://github.com/Tencent/AngelSpec/issues
• Pull requests: https://github.com/Tencent/AngelSpec/pulls

12. Summary

AngelSpec packages Tencent's production speculative-decoding training pipeline into an open, configuration-driven framework spanning six draft architectures and both vLLM and SGLang backends. It is most valuable for teams that need to cut LLM inference latency at scale — either by deploying its already-released draft models or by training custom ones — without building a speculative-decoding training stack from scratch.