Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity
Seonghoon Yu, Dongjun Nam, Dina Katabi, Jeany Son · 2025
Read Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity by Seonghoon Yu, Dongjun Nam, Dina Katabi, Jeany Son (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22480.txt.
Description
Seonghoon Yu and three co-authors improve knowledge distillation, where a small student model learns from a large teacher. Varied teacher views help the student, but using many teacher networks costs too much. Their method attaches several branches to one teacher, creating many views from a single source at low cost.
Two losses shape the views by their angles, spreading them evenly around the original teacher output while staying close to it. The mixed knowledge then trains the student, and theory shows the spread views lower the expected loss bound. Experiments beat an earlier augmentation method and the approach plugs into other distillation setups.
Catalog record
| CREATOR | Seonghoon Yu, Dongjun Nam, Dina Katabi, Jeany Son |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 105 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-22480