Knowledge Distillation with Training Wheels
Guanlin Liu, Anand Ramachandran, Tanmay Gangwani, Yan Fu, Abhinav Sethy, Amazon Alexa AI · 2025
Read Knowledge Distillation with Training Wheels by Guanlin Liu, Anand Ramachandran, Tanmay Gangwani, Yan Fu, Abhinav Sethy, Amazon Alexa AI (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-17717.txt.
Description
Guanlin Liu and co-authors wrote a 2025 paper on a new way to train a small language model with the help of a large one. The small model also learns when to ask for help at test time. In the standard approach, the large model is a teacher during training and gone at test time. Here the help line stays open, at a price.
The team frames the teaching relationship as a problem of value and cost, then solves it with a path-fitting technique. The framework grows to add hard limits on live help, set by rule. The small model must learn the task and learn which parts are hard, so it asks for help where the request pays most. Tests in translation and summarization tasks showed the value of the framework. Its performance reached operating points that a known fast-decoding trick cannot reach, the team says.
Catalog record
| CREATOR | Guanlin Liu, Anand Ramachandran, Tanmay Gangwani, Yan Fu, Abhinav Sethy, Amazon Alexa AI |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-17717