Linear Representations of Hierarchical Concepts in Language Models

Masaki Sakata, Benjamin Heinzerling, Takumi Ito, Sho Yokoi, Kentaro Inui · 2026

Read Linear Representations of Hierarchical Concepts in Language Models by Masaki Sakata, Benjamin Heinzerling, Takumi Ito, Sho Yokoi, Kentaro Inui (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-07886.txt.

Description

Sakata and four colleagues study how language models encode hierarchical relations like Japan within Eastern Asia within Asia. Building on linear relational concepts, they train linear maps for each depth and semantic domain and compare them to map representational differences. Unlike prior work, the analysis covers multi-token entities and cross-layer patterns. Across several domains, ranked relations prove linearly recoverable from internal patterns within a domain. The team also tests generalization to unseen data and transfer across domains.

Catalog record

CREATORMasaki Sakata, Benjamin Heinzerling, Takumi Ito, Sho Yokoi, Kentaro Inui
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44708
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.07886

Browse the full library · About the Tower of Babel