Attractor Patch Networks: Reducing Catastrophic Forgetting with Routed Low-Rank Patch Experts

Shashank · 2026

Read Attractor Patch Networks: Reducing Catastrophic Forgetting with Routed Low-Rank Patch Experts by Shashank (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-06993.txt.

Description

Shashank offers a new part to swap in for the dense feed-forward layers of transformers. Dense layers spend the same compute on each token. They also spread capacity evenly, though language contexts cluster in tight groups. And when a model keeps learning, a small update touches shared weights and disturbs old skills.

His design, Attractor Patch Networks, keeps a bank of patch experts. A router that matches by similarity picks a few patches for each token, using learned prototypes. Each picked patch sends out a low-rank residual update tied to a compact code. In tests on character-level text, scores nearly matched a dense baseline. Yet old-domain retention was over two and a half times better, and new-domain adaptation nearly three times better, than global fine-tuning.

Catalog record

CREATORShashank
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-06993

Browse the full library · About the Tower of Babel