Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding

Jinze Li, Yixing Xu, Haiduo Huang, Xuanwu Yin, Dong Li, Edith C.H. Ngai ∗, Emad Barsoum · 2025

Read Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding by Jinze Li, Yixing Xu, Haiduo Huang, Xuanwu Yin, Dong Li, Edith C.H. Ngai ∗, Emad Barsoum (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10135.txt.

Description

Jinze Li and six co-authors present Gumiho, a design that speeds up text generation by guessing ahead, in this 2025 arXiv paper. In speculative decoding, a small draft model proposes a run of tokens and the big model checks them, accepting the ones that line up. Most designs treat every guessed token as equal, but the authors prove the early guesses matter far more, since one bad early token voids the rest.

Gumiho therefore spends its strength up front: a full transformer drafts the first tokens one by one, while light, simple heads guess the later tokens all at once. The split gives better accuracy where it counts and speed where it is safe. Tests against rival designs showed the hybrid ahead overall.

Catalog record

CREATORJinze Li, Yixing Xu, Haiduo Huang, Xuanwu Yin, Dong Li, Edith C.H. Ngai ∗, Emad Barsoum
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-10135

Browse the full library · About the Tower of Babel