Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architectures
Francesco Cagnetta, Alessandro Favero, Antonio Sclocchi, Matthieu Wyart · 2025
Read Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architectures by Francesco Cagnetta, Alessandro Favero, Antonio Sclocchi, Matthieu Wyart (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-07070.txt.
Description
Language has floors within floors: sounds build words, words build phrases, phrases build meaning. Francesco Cagnetta and co-authors ask how neural networks climb those floors, using a synthetic language generator whose grammar is hierarchical by construction and simple enough for exact theory. Their earlier account says models learn structure layer by layer, driven by correlations in the data. Here they add architecture to the story.
Convolutional networks, whose local, shared filters mirror how the synthetic sentences are generated, improve faster as data grows. Transformers, which attend everywhere at once, scale more slowly on the same task. The authors argue scaling laws are born from the fit between a model's shape and the data's hidden structure.
Catalog record
| CREATOR | Francesco Cagnetta, Alessandro Favero, Antonio Sclocchi, Matthieu Wyart |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-07070