Attractor Patch Networks: Reducing Catastrophic Forgetting with Routed Low-Rank Patch Experts
Shashank · 2026
Read Attractor Patch Networks: Reducing Catastrophic Forgetting with Routed Low-Rank Patch Experts by Shashank (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-06993.txt.
Description
Shashank offers a new part to swap in for the dense feed-forward layers of transformers. Dense layers spend the same compute on each token. They also spread capacity evenly, though language contexts cluster in tight groups. And when a model keeps learning, a small update touches shared weights and disturbs old skills.
His design, Attractor Patch Networks, keeps a bank of patch experts. A router that matches by similarity picks a few patches for each token, using learned prototypes. Each picked patch sends out a low-rank residual update tied to a compact code. In tests on character-level text, scores nearly matched a dense baseline. Yet old-domain retention was over two and a half times better, and new-domain adaptation nearly three times better, than global fine-tuning.
Catalog record
| CREATOR | Shashank |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-06993