Linear Representations of Hierarchical Concepts in Language Models
Masaki Sakata, Benjamin Heinzerling, Takumi Ito, Sho Yokoi, Kentaro Inui · 2026
Read Linear Representations of Hierarchical Concepts in Language Models by Masaki Sakata, Benjamin Heinzerling, Takumi Ito, Sho Yokoi, Kentaro Inui (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-07886.txt.
Description
Sakata and four colleagues study how language models encode hierarchical relations like Japan within Eastern Asia within Asia. Building on linear relational concepts, they train linear maps for each depth and semantic domain and compare them to map representational differences. Unlike prior work, the analysis covers multi-token entities and cross-layer patterns. Across several domains, ranked relations prove linearly recoverable from internal patterns within a domain. The team also tests generalization to unseen data and transfer across domains.
Catalog record
| CREATOR | Masaki Sakata, Benjamin Heinzerling, Takumi Ito, Sho Yokoi, Kentaro Inui |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44708 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.07886