Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity

Seonghoon Yu, Dongjun Nam, Dina Katabi, Jeany Son · 2025

Read Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity by Seonghoon Yu, Dongjun Nam, Dina Katabi, Jeany Son (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22480.txt.

Description

Seonghoon Yu and three co-authors improve knowledge distillation, where a small student model learns from a large teacher. Varied teacher views help the student, but using many teacher networks costs too much. Their method attaches several branches to one teacher, creating many views from a single source at low cost.

Two losses shape the views by their angles, spreading them evenly around the original teacher output while staying close to it. The mixed knowledge then trains the student, and theory shows the spread views lower the expected loss bound. Experiments beat an earlier augmentation method and the approach plugs into other distillation setups.

Catalog record

CREATORSeonghoon Yu, Dongjun Nam, Dina Katabi, Jeany Son
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE105 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-22480

Browse the full library · About the Tower of Babel