Knowledge Distillation with Training Wheels

Guanlin Liu, Anand Ramachandran, Tanmay Gangwani, Yan Fu, Abhinav Sethy, Amazon Alexa AI · 2025

Read Knowledge Distillation with Training Wheels by Guanlin Liu, Anand Ramachandran, Tanmay Gangwani, Yan Fu, Abhinav Sethy, Amazon Alexa AI (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-17717.txt.

Description

Guanlin Liu and co-authors wrote a 2025 paper on a new way to train a small language model with the help of a large one. The small model also learns when to ask for help at test time. In the standard approach, the large model is a teacher during training and gone at test time. Here the help line stays open, at a price.

The team frames the teaching relationship as a problem of value and cost, then solves it with a path-fitting technique. The framework grows to add hard limits on live help, set by rule. The small model must learn the task and learn which parts are hard, so it asks for help where the request pays most. Tests in translation and summarization tasks showed the value of the framework. Its performance reached operating points that a known fast-decoding trick cannot reach, the team says.

Catalog record

CREATORGuanlin Liu, Anand Ramachandran, Tanmay Gangwani, Yan Fu, Abhinav Sethy, Amazon Alexa AI
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-17717

Browse the full library · About the Tower of Babel