BRIDGE: Budget-aware Reasoning via Intermediate Distillation with Guided Examples

Xuan-An Le, Minh-Nam Tran, Son Nguyen · 2025

Read BRIDGE: Budget-aware Reasoning via Intermediate Distillation with Guided Examples by Xuan-An Le, Minh-Nam Tran, Son Nguyen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-20403.txt.

Description

Xuan-An Le, Minh-Nam Tran, and Son Nguyen shrink frontier language models into tiny ones without paying frontier prices. Direct distillation from a huge teacher to a sub-billion-parameter student fails twice over: the capacity gap is enormous, and teacher queries are expensive. Their approach, BRIDGE, inserts a mid-sized teaching assistant between them.

First the assistant learns from the teacher on a small, carefully chosen slice of data, picked for difficulty and diversity at zero query cost. Then the assistant freely generates rationales for the full dataset to train the student, after an instruction-tuning warm-up. Across medical, legal, and financial benchmarks, students gain substantially and close much of the gap to the teacher. BRIDGE uses a tenth of the teacher queries of direct distillation.

Catalog record

CREATORXuan-An Le, Minh-Nam Tran, Son Nguyen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE92 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-20403

Browse the full library · About the Tower of Babel