A Multi-Model Adaptation of Speculative Decoding for Classification

Somnath Roy, Padharthi Sreekar, Srivatsa Narasimha, Anubhav Anand · 2025

Read A Multi-Model Adaptation of Speculative Decoding for Classification by Somnath Roy, Padharthi Sreekar, Srivatsa Narasimha, Anubhav Anand (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18076.txt.

Description

Somnath Roy and co-authors adapt, in twenty twenty-five, speculative decoding from text generation to classification. Speculative decoding uses small draft models to propose answers that a big model checks. Their framework flips that idea: up to three lightweight worker models predict class labels independently. When a majority agree, the label is accepted, skipping the expensive judge model. When they disagree, the judge decides. This balances efficiency and accuracy, they argue.

They found small three-billion-parameter chat models align with judges about as well as larger seven-billion models after tuning. The best small worker pair agreed with judges on eighty to eighty-three percent of sentiment cases. Harder ticket-similarity cases ranged from fifty to eighty percent. The design confines costly judging to hard cases.

Catalog record

CREATORSomnath Roy, Padharthi Sreekar, Srivatsa Narasimha, Anubhav Anand
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-18076

Browse the full library · About the Tower of Babel