Constructing Efficient Fact-Storing MLPs for Transformers

Owen Dugan, Roberto Garcia, Ronny Junkins, Jerry Liu, [0.3ex] Dylan Zinsley, Sabri Eyuboglu, Atri Rudra, Chris Ré, [1.5ex] Computer Science Department, Stanford University · 2025

Read Constructing Efficient Fact-Storing MLPs for Transformers by Owen Dugan, Roberto Garcia, Ronny Junkins, Jerry Liu, [0.3ex] Dylan Zinsley, Sabri Eyuboglu, Atri Rudra, Chris Ré, [1.5ex] Computer Science Department, Stanford University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-00207.txt.

Introduction

Language models store facts inside their multilayer perceptron layers, arranged like key-value pairs. Owen Dugan and co-authors, including Chris Ré, study how to construct efficient fact-storing perceptrons for transformers in 2025. Earlier work proposed explicit weight constructions for such storage.

Understanding the construction improves knowledge of how models hold factual information in their parameters. Efficient storage designs could make models smaller for the same knowledge, or reveal how to edit facts deliberately. The work sits between theory and practice, treating a trained network as engineered memory rather than a black box. Explicit constructions complement statistical arguments about what trained networks probably do. They also show how much memory a given set of facts requires.

Catalog record

CREATOROwen Dugan, Roberto Garcia, Ronny Junkins, Jerry Liu, [0.3ex] Dylan Zinsley, Sabri Eyuboglu, Atri Rudra, Chris Ré, [1.5ex] Computer Science Department, Stanford University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE250 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-00207

Browse the full library · About the Tower of Babel