A Technical Study into 0.5B Reasoning Language Models

Xialie Zhuang, Peixian Ma, Zhikai Jia, Shiwei Liu, Zheng Cao · 2025

Read A Technical Study into 0.5B Reasoning Language Models by Xialie Zhuang, Peixian Ma, Zhikai Jia, Shiwei Liu, Zheng Cao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-13404.txt.

Description

Xialie Zhuang and co-authors test how to teach very small language models, half a billion parameters, to reason, in a paper from 2025. Tiny models are cheap, private, and light on energy, but their small capacity strains at math and multi-step logic. The team compared training recipes head to head.

They tried supervised fine-tuning on worked examples, distillation from a bigger teacher, reinforcement learning from feedback, and hybrids of the three. Extensive experiments measured which mixes actually lift reasoning scores. The paper distills the runs into practical guidance: which pipeline fits which budget, and where the small model's ceiling sits. The authors aim the advice at builders who need reasoning on modest hardware.

Catalog record

CREATORXialie Zhuang, Peixian Ma, Zhikai Jia, Shiwei Liu, Zheng Cao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-13404

Browse the full library · About the Tower of Babel