RMAAT: Astrocyte-Inspired Memory Compression and Replay for Efficient Long-Context Transformers
Md Zesun Ahmed Mia, Malyaban Bal, Abhronil Sengupta · 2026
Read RMAAT: Astrocyte-Inspired Memory Compression and Replay for Efficient Long-Context Transformers by Md Zesun Ahmed Mia, Malyaban Bal, Abhronil Sengupta (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-00426.txt.
Description
Md Zesun Ahmed Mia, Malyaban Bal, and Abhronil Sengupta borrow ideas from astrocytes, the glial cells that support memory in the brain, to make Transformer models handle longer texts. Standard attention grows in cost very fast as sequences lengthen. Their design, RMAAT, processes text in segments and carries context forward in special memory tokens.
A retention factor modeled on long-term changes in astrocytes controls how those memory tokens are compressed. Inside each segment, a cheaper attention step mimics short-term astrocyte effects. A new training method replays stored memories to save memory during learning. Tests on long-context benchmarks are used to judge whether the brain-inspired design keeps accuracy while cutting the cost of long sequences.
Catalog record
| CREATOR | Md Zesun Ahmed Mia, Malyaban Bal, Abhronil Sengupta |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-00426