Reward-Shifted Speculative Sampling Is An Efficient Test-Time Weak-to-Strong Aligner

Bolian Li, Yanran Wu, Xinyu Luo, Ruqi Zhang · 2025

Read Reward-Shifted Speculative Sampling Is An Efficient Test-Time Weak-to-Strong Aligner by Bolian Li, Yanran Wu, Xinyu Luo, Ruqi Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15044.txt.

Description

Bolian Li and co-authors speed up the tuning of language models at test time in a 2025 paper. Tuning a model's answers to human taste usually costs heavy compute while it runs. A trick called speculative sampling uses a small draft model to guess ahead, and the team adapts that trick for taste tuning.

In their scheme, the small draft model is the one tuned to human taste, while the big model stays as is. A shifted acceptance rule lets the pair recover, in effect, the answer a fully tuned model would give. Tests in weak-to-strong tuning scored higher on reward measures at far lower cost than prior methods.

Catalog record

CREATORBolian Li, Yanran Wu, Xinyu Luo, Ruqi Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-15044

Browse the full library · About the Tower of Babel