BRIDGE: Budget-aware Reasoning via Intermediate Distillation with Guided Examples
Xuan-An Le, Minh-Nam Tran, Son Nguyen · 2025
Read BRIDGE: Budget-aware Reasoning via Intermediate Distillation with Guided Examples by Xuan-An Le, Minh-Nam Tran, Son Nguyen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-20403.txt.
Description
Xuan-An Le, Minh-Nam Tran, and Son Nguyen shrink frontier language models into tiny ones without paying frontier prices. Direct distillation from a huge teacher to a sub-billion-parameter student fails twice over: the capacity gap is enormous, and teacher queries are expensive. Their approach, BRIDGE, inserts a mid-sized teaching assistant between them.
First the assistant learns from the teacher on a small, carefully chosen slice of data, picked for difficulty and diversity at zero query cost. Then the assistant freely generates rationales for the full dataset to train the student, after an instruction-tuning warm-up. Across medical, legal, and financial benchmarks, students gain substantially and close much of the gap to the teacher. BRIDGE uses a tenth of the teacher queries of direct distillation.
Catalog record
| CREATOR | Xuan-An Le, Minh-Nam Tran, Son Nguyen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 92 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-20403