Listen and Chant Before You Read: The Ladder of Beauty in LM Pre-Training
Yoshinori Nomura · 2026
Read Listen and Chant Before You Read: The Ladder of Beauty in LM Pre-Training by Yoshinori Nomura (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21265.txt.
Description
Yoshinori Nomura shows that pre-training a transformer on music before language speeds up language learning. A developmental pipeline from piano performances through poetry to prose cuts perplexity by 17.5 percent over random initialization, with music and poetry improving different model parts. The edge persists rather than fading: at the smallest hidden size tested, a gap remains at plateau across seeds. Real music matches synthetic patterns' transfer ceiling with one-third the data, pointing to capacity-dependent data curation.
Catalog record
| CREATOR | Yoshinori Nomura |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47669 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.21265