Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning

Hien Dang, Pratik Patil, Alessandro Rinaldo · 2026

Read Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning by Hien Dang, Pratik Patil, Alessandro Rinaldo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-17565.txt.

Description

Hien Dang led a three-author team behind a 2026 arXiv paper on self-distillation in ridge regression. Self-distillation retrains a student on true labels mixed with the teacher's predictions, using the same architecture and data. It often improves generalization, but formal guarantees have stayed limited. The paper studies the unconstrained setting, where the mixing weight may fall outside the unit interval.

Conditioning only on the training data, and assuming nothing about distributions, the authors prove the optimally mixed student strictly improves on the ridge teacher wherever teacher risk is nonstationary. They give a closed-form optimal weight and a sign rule tied to the slope of teacher risk. Exact deterministic equivalents quantify the gain as sample and feature sizes grow proportionally. A consistent one-shot tuning method estimates the optimum without grid search, sample splitting, or refitting. Tests on real data and pretrained features support the theory.

Catalog record

CREATORHien Dang, Pratik Patil, Alessandro Rinaldo
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE211 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-17565

Browse the full library · About the Tower of Babel