In Good GRACEs: Principled Teacher Selection for Knowledge Distillation
Abhishek Panigrahi Bingbin Liu Sadhika Malladi Sham Kakade Surbhi Goel · 2025
Read In Good GRACEs: Principled Teacher Selection for Knowledge Distillation by Abhishek Panigrahi Bingbin Liu Sadhika Malladi Sham Kakade Surbhi Goel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-02833.txt.
Introduction
Abhishek Panigrahi, Bingbin Liu, Sadhika Malladi, Sham Kakade, and Surbhi Goel tackle teacher choice in knowledge distillation, where a big model generates data to train a smaller one. Picking the best teacher for a given student and task usually means costly trial and error. Their 2025 paper proposes GRACE, a light score of how well a teacher will suit a student.
GRACE reads properties of the student's gradients and needs no verifier, teacher internals, or test data. It ties, through information theory, to the stability of gradient methods, which governs how well students generalize. On GSM8K and MATH, GRACE matched later student scores with up to 86 percent rank correlation. Students trained on the GRACE-picked teacher gained up to 7.4 percent over those given the strongest teacher by default.
Catalog record
| CREATOR | Abhishek Panigrahi Bingbin Liu Sadhika Malladi Sham Kakade Surbhi Goel |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 116 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-02833
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
In Good GRACEs: Principled Teacher Selection for Knowledge Distillation Abhishek Panigrahi Bingbin Liu Sadhika Malladi Sham Kakade Surbhi Goel Abstract Knowledge distillation is an efficient strategy to use data generated by large “teacher” language models to train smaller capable “student” models, but selecting the optimal teacher for a specific student-task combination requires expensive trial-and-error. We propose a lightweight score called GRACE to quantify how effective a teacher will be for post-training a student model. GRACE measures distributional properties of the student’s gradients without access to a verifier, teacher logits, teacher internals, or test data. From an information-theoretic perspective, GRACE connects to leave-one-out stability of gradient-based algorithms, which controls the generalization performance of the distilled students. On GSM8K and MATH, GRACE correlates strongly (up to 86 % 86\% Spearman correlation) with the performance of the distilled LLaMA and OLMo students. In particular, training a student using the GRACE -selected teacher can improve the performance by up to 7.4 % 7.4\% over naively using the best-performing teacher. Further, GRACE can provide guidance on crucial design choices in distillation, including (1) the best temperature to use when generating from the teacher, (2) the best teacher to use given a size constraint, and (3) the best teacher to use within a specific model family. Altogether, our findings demonstrate that GRACE can efficiently and effectively identify a strongly compatible teacher for a given student and provide fine-grained guidance on how to perform distillation. † † footnotetext: Correspondence to: [email protected] , [email protected] 1 Introduction Distillation is an efficient and effective method to produce capable small models from existing, powerful teacher models. In this work, we focus on the specific case of training autoregressive language models on text generated by a teacher model. It is difficult to select the right teacher for a given student and task: a counterintuitive fact is that a stronger-performing model is not always a better teacher , which has been observed in classic classification/regression settings ( 33 ; 16 ; 36 ) and more recently in the context of language models ( 57 ; 56 ; 38 ; 42 ) . Given the large number of available models as potential teachers, the current approach of guess-and-check is costly, because it requires collecting generations from a capable teacher and subsequently training a student on those generations. Additionally, the specific hyperparameters used in both phases can dramatically affect the final performance of the student, underscoring the need for careful, repeated testing to select the right teacher. As such, the current work seeks to address the following question: Given a pool of candidates, can we efficiently identify the best teacher for a given student and task? Figure 1 : GRACE correlates with student performance after distillation on math-related reasoning tasks. We evaluate LLaMA-1B and LLaMA-3B students on GSM8K and MATH respectively with 15 teachers across LLaMA, Gemma, Qwen, OLMo, and Phi families. (Left) GRACE shows the strongest correlation with student accuracy among four scores—teacher performance, student’s loss (before training) on the teacher generations, G-Vendi , and GRACE . (Right) GRACE reliably selects near-optimal teachers within each teacher family , measured by its small teacher-student regret , which is the absolute gap in final performance between the best overall student and that obtained from the teacher chosen by each score. Performance is measured by average accuracy over 16 generations per prompt. We argue that identifying a good teacher requires jointly considering the teacher and the student. To this end, we study the teacher’s influence on the student’s optimization process by analyzing the student’s gradients. We propose a score, GRACE (GRAdient Cross-validation Evaluation), which captures the distributional properties of the student’s gradients on a small set of teacher-generated data, enabling efficient and effective identification of the most compatible teacher ( Section 2.2 ). Inspired by prior data selection and distillation works, GRACE unifies data diversity and teacher-student alignment desiderata into a single score. Specifically, GRACE leverages a cross-validation structure, where it takes gradients from one split of data and measures their magnitudes re-weighted under the spectrum of the other split. The gradient spectrum reflects data diversity, while the gradient norm encodes teacher-student alignment. This cross-validation structure also allows us to draw a natural connection to conditional mutual information-based generalization bounds ( 48 ; 41 ) , providing insight into why GRACE works ( Lemma 1 ). Importantly, GRACE is efficient to compute and does not rely on access to an external verifier, teacher logits, teacher representations, or test data. We conduct thorough experiments to verify that the GRACE scores of teachers accurately reflect the student’s performance when trained on those teachers. We focus on math-related datasets, namely GSM8K ( 8 ) and MATH ( 17 ) , because the broad community interest in mathematical reasoning has led to the development of a large, diverse set of teachers that are readily available and suitable for distillation. We train LLaMA-1B-Base, OLMo-1B-Base and Gemma-2B-Base (for GSM8K) as well as LLaMA-3B-Base (for MATH) using generations sampled from 15 candidate teachers drawn from the LLaMA ( 51 ) , OLMo ( 35 ) , Qwen ( 40 ) , Gemma ( 50 ) , and Phi ( 1 ) families. Our results show that: • GRACE correlates strongly with the student’s distillation performance ( Figure 1 , left), outperforming strong baselines such as G-Vendi ( 22 ) . • GRACE identifies teachers that are strongly compatible to a given student. Selecting teachers using GRACE yields 7\% and 2\% improvement in student performance compared to using the best-performing teacher, on GSM8K and MATH respectively. • GRACE offers actionable insights to practitioners. It helps identify close to 1) optimal generation temperature for a given teacher model, 2) best teacher within a size constraint, and 3) best teacher within a model family. Given the limited research on teacher selection in LLM distillation, our findings indicate that GRACE improves performance and offers reliable guidance for effective distillation. 2 GRACE: Gradient Cross-Validation Evaluation We consider the case of using distillation to fine-tune a pre-trained student model to solve specific downstream tasks. For each of the N prompts \mathbf{x}\in\mathcal{X} , we autoregressively generate M responses y_{1},...,y_{M} from a teacher distribution \pi_{\cal T} . This distribution encodes the temperature it may be sampled at from the teacher as well. We then fine-tune the pre-trained student with the standard autoregressive cross-entropy objective \mathcal{L} on a dataset \mathcal{D}^{\text{distill}}_{\cal T} containing N\times M teacher generations. In contrast to logit-based distillation, learning from generations permits distillation across architectures and in cases where the teacher’s logits are not available. We measure the performance of students and teachers as the average accuracy of k sampled responses for a given prompt (i.e., average-at-k). We will use \pi_{\cal S} to denote the pre-trained student, and refer to its parameters as \Theta_{\cal S}\in\mathbb{R}^{D} when necessary. 2.1 Gradient-Based Scores The problem of selecting a teacher for distillation is closely connected to the well-studied field of data selection: choosing the best teacher based on its generations can be viewed as selecting the best subset from the union of all teachers’ generations, with the constraint that each subset must come from a single teacher. For language models, many successful data selection methods ( 39 ; 52 ; 12 ) rely on