Distilling LLM Reasoning into Graph of Concept Predictors

Ziyang Yu, Liang Zhao · 2026

Read Distilling LLM Reasoning into Graph of Concept Predictors by Ziyang Yu, Liang Zhao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-03006.txt.

Description

Ziyang Yu and 1 co-author distill a large model's reasoning into a graph of small predictors in a 2026 paper. Running large language models for classification at scale is slow and costly. Standard distillation copies only final labels, discarding the teacher's intermediate reasoning and hiding where student errors come from in the training data.

Their framework, GCP, asks the teacher for reasoning as a graph of intermediate concepts. The student mirrors that graph with its own concept predictors. A graph-aware strategy picks which examples to label next, weighing concept uncertainty, diversity, and centrality. Targeted retraining updates only the most influential concept modules. Tests on eight classification benchmarks showed better performance under limited labeling budgets, with more interpretable training dynamics.

Catalog record

CREATORZiyang Yu, Liang Zhao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-03006

Browse the full library · About the Tower of Babel