hiw3c 日报精选
arXiv cs.AI/CL/LG 网页快照

窄规格:推测解码中隐藏宽度无需与目标值匹配

原文标题 · NarrowSpec: Draft Hidden Width Need Not Match the Target in Speculative Decoding

作者: Alagappan Valliappan, Ankit More, Dheevatsa Mudigere | 推测解码被广泛用于加速大语言模型(LLM)的生成过程,其中小型草稿模型提出的标记在目标模型的一次前向传播中得到验证。读取目标模型隐藏状态的草稿模型,如DFlash和EAGLE-3,通常继承目标模型的隐藏宽度。我们表明,该草稿的整个残差流可运行于该宽度的一半或四分之一处。

来源:arXiv cs.AI/CL/LG · arxiv.org

正文为英文,可一键机器翻译(仅首次需要等待)

Computer Science > Computation and Language

Title: NarrowSpec: Draft Hidden Width Need Not Match the Target in Speculative Decoding

Abstract: Speculative decoding is widely used to speed up generation in large language models (LLMs), with a small draft model proposing tokens that one forward pass of the target model verifies. Draft models that read the target's hidden states, such as DFlash and EAGLE-3, typically inherit the target's hidden width. We show that the draft's whole residual stream can run at half or a quarter of that width while largely preserving acceptance. For DFlash, which borrows the target's vocabulary projection, the trained output adapter folds into it, giving one narrower projection with logits unchanged up to floating-point rounding. We test two drafter families, DFlash and EAGLE-3, on two targets, Qwen3-8B (NVFP4) and GPT-OSS-20B (MXFP4), in SGLang on a B200 and a DGX Spark (GB10). At batch size 1 and temperature 0, narrowing the draft to half or a quarter width speeds up end-to-end decoding by 1.06-1.12x on B200 and 1.13-1.44x on GB10. The speedup holds at larger batch sizes. From batch 32 to the largest batch each GPU fits, narrowing reaches up to 1.21x on both GPUs, and no narrow draft is slower than full width.

Submission history

Access Paper:

Additional Features

Current browse context:

References & Citations

BibTeX formatted citation

Bookmark

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .