The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Level
Jeremy Herbst, Jae Hee Lee, Stefan Wermter · 2026
Read The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Level by Jeremy Herbst, Jae Hee Lee, Stefan Wermter (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-02178.txt.
Introduction
Herbst, Wermter and Lee ask whether mixture of experts models are easier to interpret than dense networks. Probing with sparse methods, they find expert neurons less polysemantic than dense feed forward neurons, with the gap widening as routing grows sparser. That points to the expert, not the neuron, as the better unit of analysis. Auto interpreting hundreds of experts shows they are fine grained task experts, not broad domain specialists or mere token processors.
Catalog record
| CREATOR | Jeremy Herbst, Jae Hee Lee, Stefan Wermter |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 181374 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.02178