SmolTulu: Higher Learning Rate to Batch Size Ratios Can Lead to Better Reasoning in SLMs
Sultan Alrashed · 2024
Read SmolTulu: Higher Learning Rate to Batch Size Ratios Can Lead to Better Reasoning in SLMs by Sultan Alrashed (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08347.txt.
Description
Sultan Alrashed reports how training ratios shape the reasoning of small language models in this 2024 paper. The work adapts a known post-training recipe to a 1.7-billion-parameter base model. Experiments with a smaller test model explored how learning rate and batch size interact.
A clear split emerged: reasoning tasks improved with higher learning-rate-to-batch ratios, while pattern tasks preferred lower ones. Guided by that split, the tuned model reached top scores among small models on instruction following. It also posted strong gains on grade-school math and science questions. The author releases the model, recipes, and ablation studies. He argues careful tuning of training dynamics can narrow the gap between small and large models.
Catalog record
| CREATOR | Sultan Alrashed |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-08347