Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
Jinze Li, Yixing Xu, Haiduo Huang, Xuanwu Yin, Dong Li, Edith C.H. Ngai ∗, Emad Barsoum · 2025
Read Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding by Jinze Li, Yixing Xu, Haiduo Huang, Xuanwu Yin, Dong Li, Edith C.H. Ngai ∗, Emad Barsoum (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10135.txt.
Description
Jinze Li and six co-authors present Gumiho, a design that speeds up text generation by guessing ahead, in this 2025 arXiv paper. In speculative decoding, a small draft model proposes a run of tokens and the big model checks them, accepting the ones that line up. Most designs treat every guessed token as equal, but the authors prove the early guesses matter far more, since one bad early token voids the rest.
Gumiho therefore spends its strength up front: a full transformer drafts the first tokens one by one, while light, simple heads guess the later tokens all at once. The split gives better accuracy where it counts and speed where it is safe. Tests against rival designs showed the hybrid ahead overall.
Catalog record
| CREATOR | Jinze Li, Yixing Xu, Haiduo Huang, Xuanwu Yin, Dong Li, Edith C.H. Ngai ∗, Emad Barsoum |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-10135