A Unified Revisit of Temperature in Classification-Based Knowledge Distillation
Logan Frank Jim Davis · 2026
Read A Unified Revisit of Temperature in Classification-Based Knowledge Distillation by Logan Frank Jim Davis (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-02430.txt.
Description
Logan Frank and Jim Davis revisit one key dial in knowledge distillation in this 2026 paper. Distillation trains a small student model to copy a larger teacher. A temperature dial softens the teacher's outputs, exposing relations between classes. Picking that dial well is poorly understood, and grid search wastes time.
The authors argue temperature is tightly linked to other training choices. Optimizer, teacher pretraining, and fine-tuning all interact with the best setting. Their unified study maps those interactions in a systematic way. Certain common situations shift temperature choice in pronounced ways, they find. Values borrowed from other setups can quietly hurt student performance. The paper turns those patterns into practical guidance for training students.
Catalog record
| CREATOR | Logan Frank Jim Davis |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35971 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.02430