Listen and Chant Before You Read: The Ladder of Beauty in LM Pre-Training

Yoshinori Nomura · 2026

Read Listen and Chant Before You Read: The Ladder of Beauty in LM Pre-Training by Yoshinori Nomura (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21265.txt.

Description

Yoshinori Nomura shows that pre-training a transformer on music before language speeds up language learning. A developmental pipeline from piano performances through poetry to prose cuts perplexity by 17.5 percent over random initialization, with music and poetry improving different model parts. The edge persists rather than fading: at the smallest hidden size tested, a gap remains at plateau across seeds. Real music matches synthetic patterns' transfer ceiling with one-third the data, pointing to capacity-dependent data curation.

Catalog record

CREATORYoshinori Nomura
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47669
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.21265

Browse the full library · About the Tower of Babel