STree: Speculative Tree Decoding for Hybrid State-Space Models

Yangchao Wu, Zongyue Qin, Alex Wong, Stefano Soatto · 2025

Read STree: Speculative Tree Decoding for Hybrid State-Space Models by Yangchao Wu, Zongyue Qin, Alex Wong, Stefano Soatto (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14969.txt.

Description

Wu, Qin, Wong, and Soatto present a study from twenty twenty-five on speeding up a lean type of language model. Standard chat models gain speed by drafting several words ahead and checking them at once. The lean models in focus keep a compact state of past text, so they are already fast. Draft checks for them had not used tree drafts, which branch into many word paths. Their method is the first scalable way to check tree drafts in such models. It leans on how state shift tables build up over time.

Their code is shaped to fit the hardware it runs on. It beat plain draft checking on three test sets, even with a basic draft model. The authors say more speed gains remain open. The work also suits mixed models that blend both types.

Catalog record

CREATORYangchao Wu, Zongyue Qin, Alex Wong, Stefano Soatto
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-14969

Browse the full library · About the Tower of Babel