A Technical Study into 0.5B Reasoning Language Models
Xialie Zhuang, Peixian Ma, Zhikai Jia, Shiwei Liu, Zheng Cao · 2025
Read A Technical Study into 0.5B Reasoning Language Models by Xialie Zhuang, Peixian Ma, Zhikai Jia, Shiwei Liu, Zheng Cao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-13404.txt.
Description
Xialie Zhuang and co-authors test how to teach very small language models, half a billion parameters, to reason, in a paper from 2025. Tiny models are cheap, private, and light on energy, but their small capacity strains at math and multi-step logic. The team compared training recipes head to head.
They tried supervised fine-tuning on worked examples, distillation from a bigger teacher, reinforcement learning from feedback, and hybrids of the three. Extensive experiments measured which mixes actually lift reasoning scores. The paper distills the runs into practical guidance: which pipeline fits which budget, and where the small model's ceiling sits. The authors aim the advice at builders who need reasoning on modest hardware.
Catalog record
| CREATOR | Xialie Zhuang, Peixian Ma, Zhikai Jia, Shiwei Liu, Zheng Cao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-13404