Distilling LLM Reasoning into Graph of Concept Predictors
Ziyang Yu, Liang Zhao · 2026
Read Distilling LLM Reasoning into Graph of Concept Predictors by Ziyang Yu, Liang Zhao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-03006.txt.
Description
Ziyang Yu and 1 co-author distill a large model's reasoning into a graph of small predictors in a 2026 paper. Running large language models for classification at scale is slow and costly. Standard distillation copies only final labels, discarding the teacher's intermediate reasoning and hiding where student errors come from in the training data.
Their framework, GCP, asks the teacher for reasoning as a graph of intermediate concepts. The student mirrors that graph with its own concept predictors. A graph-aware strategy picks which examples to label next, weighing concept uncertainty, diversity, and centrality. Targeted retraining updates only the most influential concept modules. Tests on eight classification benchmarks showed better performance under limited labeling budgets, with more interpretable training dynamics.
Catalog record
| CREATOR | Ziyang Yu, Liang Zhao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-03006