Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architectures

Francesco Cagnetta, Alessandro Favero, Antonio Sclocchi, Matthieu Wyart · 2025

Read Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architectures by Francesco Cagnetta, Alessandro Favero, Antonio Sclocchi, Matthieu Wyart (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-07070.txt.

Description

Language has floors within floors: sounds build words, words build phrases, phrases build meaning. Francesco Cagnetta and co-authors ask how neural networks climb those floors, using a synthetic language generator whose grammar is hierarchical by construction and simple enough for exact theory. Their earlier account says models learn structure layer by layer, driven by correlations in the data. Here they add architecture to the story.

Convolutional networks, whose local, shared filters mirror how the synthetic sentences are generated, improve faster as data grows. Transformers, which attend everywhere at once, scale more slowly on the same task. The authors argue scaling laws are born from the fit between a model's shape and the data's hidden structure.

Catalog record

CREATORFrancesco Cagnetta, Alessandro Favero, Antonio Sclocchi, Matthieu Wyart
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-07070

Browse the full library · About the Tower of Babel