An evolutionary perspective on modes of learning in Transformers

Alexander Y. Ku, Thomas L. Griffiths, Stephanie C.Y. Chan · 2025

Read An evolutionary perspective on modes of learning in Transformers by Alexander Y. Ku, Thomas L. Griffiths, Stephanie C.Y. Chan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-09855.txt.

Description

Alexander Ku, Thomas Griffiths, and Stephanie Chan compare two ways a Transformer model learns, through a lens from biology, in a paper from twenty twenty-five. These models can slowly bake knowledge into their weights, or use the prompt in front of them to adapt at once. Biology makes a like split between slow genetic change and flexible traits within one life.

The authors vary how stable the test setting is and how reliable its cues are. Stable settings favor learning in the weights, often with a sharp switch. Shifting settings with good cues favor learning from context. Shifts between the two modes also appear during training. What drives a shift is how good the mode can get, and how costly it is to acquire.

Catalog record

CREATORAlexander Y. Ku, Thomas L. Griffiths, Stephanie C.Y. Chan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-09855

Browse the full library · About the Tower of Babel