Superposition in Transformers: A Novel Way of Building Mixture of Experts
Ayoub Ben Chaliah, Hela Dellagi · 2025
Read Superposition in Transformers: A Novel Way of Building Mixture of Experts by Ayoub Ben Chaliah, Hela Dellagi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-00530.txt.
Description
Ayoub Ben Chaliah and Hela Dellagi stack model states inside one Transformer to fight forgetting, in work whose head is dated 2024. Fine-tuning a large language model on new tasks often wipes what it knew. Their design blends the inner states of a base model and a tuned twin in one shared space. Autoencoders rebuild those states based on the input at hand.
Blend weights from smooth curve tools set how much of each state shows. The tuned skill can thus be added in a compact form. States can also swap live as the model runs. Old skills stay in place while new domain skill joins.
Catalog record
| CREATOR | Ayoub Ben Chaliah, Hela Dellagi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-00530