Reward-Shifted Speculative Sampling Is An Efficient Test-Time Weak-to-Strong Aligner
Bolian Li, Yanran Wu, Xinyu Luo, Ruqi Zhang · 2025
Read Reward-Shifted Speculative Sampling Is An Efficient Test-Time Weak-to-Strong Aligner by Bolian Li, Yanran Wu, Xinyu Luo, Ruqi Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15044.txt.
Description
Bolian Li and co-authors speed up the tuning of language models at test time in a 2025 paper. Tuning a model's answers to human taste usually costs heavy compute while it runs. A trick called speculative sampling uses a small draft model to guess ahead, and the team adapts that trick for taste tuning.
In their scheme, the small draft model is the one tuned to human taste, while the big model stays as is. A shifted acceptance rule lets the pair recover, in effect, the answer a fully tuned model would give. Tests in weak-to-strong tuning scored higher on reward measures at far lower cost than prior methods.
Catalog record
| CREATOR | Bolian Li, Yanran Wu, Xinyu Luo, Ruqi Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-15044