SmolTulu: Higher Learning Rate to Batch Size Ratios Can Lead to Better Reasoning in SLMs

Sultan Alrashed · 2024

Read SmolTulu: Higher Learning Rate to Batch Size Ratios Can Lead to Better Reasoning in SLMs by Sultan Alrashed (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08347.txt.

Description

Sultan Alrashed reports how training ratios shape the reasoning of small language models in this 2024 paper. The work adapts a known post-training recipe to a 1.7-billion-parameter base model. Experiments with a smaller test model explored how learning rate and batch size interact.

A clear split emerged: reasoning tasks improved with higher learning-rate-to-batch ratios, while pattern tasks preferred lower ones. Guided by that split, the tuned model reached top scores among small models on instruction following. It also posted strong gains on grade-school math and science questions. The author releases the model, recipes, and ablation studies. He argues careful tuning of training dynamics can narrow the gap between small and large models.

Catalog record

CREATORSultan Alrashed
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-08347

Browse the full library · About the Tower of Babel