Superposition in Transformers: A Novel Way of Building Mixture of Experts

Ayoub Ben Chaliah, Hela Dellagi · 2025

Read Superposition in Transformers: A Novel Way of Building Mixture of Experts by Ayoub Ben Chaliah, Hela Dellagi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-00530.txt.

Description

Ayoub Ben Chaliah and Hela Dellagi stack model states inside one Transformer to fight forgetting, in work whose head is dated 2024. Fine-tuning a large language model on new tasks often wipes what it knew. Their design blends the inner states of a base model and a tuned twin in one shared space. Autoencoders rebuild those states based on the input at hand.

Blend weights from smooth curve tools set how much of each state shows. The tuned skill can thus be added in a compact form. States can also swap live as the model runs. Old skills stay in place while new domain skill joins.

Catalog record

CREATORAyoub Ben Chaliah, Hela Dellagi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-00530

Browse the full library · About the Tower of Babel