Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning
Hien Dang, Pratik Patil, Alessandro Rinaldo · 2026
Read Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning by Hien Dang, Pratik Patil, Alessandro Rinaldo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-17565.txt.
Description
Hien Dang led a three-author team behind a 2026 arXiv paper on self-distillation in ridge regression. Self-distillation retrains a student on true labels mixed with the teacher's predictions, using the same architecture and data. It often improves generalization, but formal guarantees have stayed limited. The paper studies the unconstrained setting, where the mixing weight may fall outside the unit interval.
Conditioning only on the training data, and assuming nothing about distributions, the authors prove the optimally mixed student strictly improves on the ridge teacher wherever teacher risk is nonstationary. They give a closed-form optimal weight and a sign rule tied to the slope of teacher risk. Exact deterministic equivalents quantify the gain as sample and feature sizes grow proportionally. A consistent one-shot tuning method estimates the optimum without grid search, sample splitting, or refitting. Tests on real data and pretrained features support the theory.
Catalog record
| CREATOR | Hien Dang, Pratik Patil, Alessandro Rinaldo |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 211 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-17565