Long-Short Chain-of-Thought Mixture Supervised Fine-Tuning Eliciting Efficient Reasoning in Large Language Models

Bin Yu, Hang Yuan, Haotian Li, Xueyin Xu, Yuliang Wei, Bailing Wang, Weizhen Qi, Kai Chen · 2025

Read Long-Short Chain-of-Thought Mixture Supervised Fine-Tuning Eliciting Efficient Reasoning in Large Language Models by Bin Yu, Hang Yuan, Haotian Li, Xueyin Xu, Yuliang Wei, Bailing Wang, Weizhen Qi, Kai Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03469.txt.

Description

Bin Yu and seven co-authors teach language models to reason in fewer words. Their paper is from two thousand twenty-five. Models trained on long, step-by-step solutions from a big teacher model tend to overthink. They write out long, rambling chains even for easy problems, which wastes time and money.

Their fix mixes long solutions with short rewrites of the same solutions. The rewrites keep the structure of the reasoning but trim the fat. Models trained on the mix scored about two percent higher on tests, on average. At the same time, their answers shrank by nearly half. The method gives smaller models real reasoning skill without the bloat. It is a simple change to a standard training recipe.

Catalog record

CREATORBin Yu, Hang Yuan, Haotian Li, Xueyin Xu, Yuliang Wei, Bailing Wang, Weizhen Qi, Kai Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-03469

Browse the full library · About the Tower of Babel