Lifelong In-Context Learning with Transformers Requires Parametric Forms of Attention

Luke McDermott, Robert W. Heath Jr, Rahul Parhi · 2026

Read Lifelong In-Context Learning with Transformers Requires Parametric Forms of Attention by Luke McDermott, Robert W. Heath Jr, Rahul Parhi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25342.txt.

Description

Agents that run for months should keep learning from experience, but a transformer's memory grows with every token it reads. Luke McDermott, Robert Heath Jr, and Rahul Parhi argue in 2026 that lifelong in context learning demands a different kind of attention. Parametric attention replaces the ever growing store of past keys and values with a small trainable module that absorbs new associations on the fly. Memory then stays constant no matter how long the agent operates, fitting a fixed hardware budget.

Surveying state space models, linear attention, test time training layers, and fast weight programmers, the authors find each current option falls short somewhere, through limited capacity or expensive updates. Rather than proposing one more mechanism, they pose a set of open questions with fresh analysis meant to steer the field. The piece is a position and agenda paper for long horizon agents.

Catalog record

CREATORLuke McDermott, Robert W. Heath Jr, Rahul Parhi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.25342

Browse the full library · About the Tower of Babel