Layer Specialization Underlying Compositional Reasoning in Transformers

Jing Liu · 2025

Read Layer Specialization Underlying Compositional Reasoning in Transformers by Jing Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17469.txt.

Description

Language models based on the transformer design can reason in steps about new sequences they never saw in training, and Jing Liu, writing alone, asks what inside the model makes that possible. Liu uses a tidy test bed called the Random Hierarchy Model, where sequences are built by recursive rules much like a grammar. Models are trained on only part of the possible sequences and then tested under four conditions, from plain memorisation to transfer across rule layers.

Scores rose steadily with task difficulty handled and with more examples given in context, though unfamiliar rule use needed far more examples. Inside the model, Liu finds that separate layers gradually specialise during training, forming ordered, hierarchical representations. That inner specialisation tracks the outer reasoning gains closely.

Catalog record

CREATORJing Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-17469

Browse the full library · About the Tower of Babel