Skill-Conditioned Gated Self-Distillation for LLM Reasoning

Jiazhen Huang, Xiao Chen, Xiao Luo, Yong Dai, Senkang Hu, Yuzhi Zhao · 2026

Read Skill-Conditioned Gated Self-Distillation for LLM Reasoning by Jiazhen Huang, Xiao Chen, Xiao Luo, Yong Dai, Senkang Hu, Yuzhi Zhao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28791.txt.

Description

Jiazhen Huang and five colleagues propose Skill-Conditioned Gated Self-Distillation, a method letting LLM reasoning improve from a skill bank rather than reference answers. Retrieved skills are compact but may mislead, so the method treats distillation as teacher validation, not blind imitation. Skill-mistake pairs feed a multi-teacher pool where skill teachers score the same student rollout. The verifier checks each teacher's polarity. A robust gated objective distills useful teacher-student disagreements, suppressing uncertain or extreme signals. It beats GRPO and rivals answer-conditioned methods on math benchmarks.

Catalog record

CREATORJiazhen Huang, Xiao Chen, Xiao Luo, Yong Dai, Senkang Hu, Yuzhi Zhao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55965
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.28791

Browse the full library · About the Tower of Babel