Learning curves theory for hierarchically compositional data with power-law distributed features

Francesco Cagnetta, Hyunmo Kang, Matthieu Wyart · 2025

Read Learning curves theory for hierarchically compositional data with power-law distributed features by Francesco Cagnetta, Hyunmo Kang, Matthieu Wyart (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-07067.txt.

Description

Francesco Cagnetta, Hyunmo Kang, and Matthieu Wyart study how learning machines improve with more data, in a 2025 theory paper. A known pattern, called a scaling law, says error falls in a steady curve as data grows. The authors ask how that curve arises when data is built in layers, the way grammar builds sentences from rules inside rules.

They study two tasks on such layered data: sorting items into classes and guessing the next word. For sorting, the curve's steepness is set by how the grammar rules are spread out, while the layered build inflates the error by a large fixed factor. For next-word guessing, the rule spread shapes only small wiggles in the curve, not its overall slope. The work ties two rival accounts of scaling laws into one picture.

Catalog record

CREATORFrancesco Cagnetta, Hyunmo Kang, Matthieu Wyart
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-07067

Browse the full library · About the Tower of Babel