The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Level

Jeremy Herbst, Jae Hee Lee, Stefan Wermter · 2026

Read The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Level by Jeremy Herbst, Jae Hee Lee, Stefan Wermter (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-02178.txt.

Introduction

Herbst, Wermter and Lee ask whether mixture of experts models are easier to interpret than dense networks. Probing with sparse methods, they find expert neurons less polysemantic than dense feed forward neurons, with the gap widening as routing grows sparser. That points to the expert, not the neuron, as the better unit of analysis. Auto interpreting hundreds of experts shows they are fine grained task experts, not broad domain specialists or mere token processors.

Catalog record

CREATORJeremy Herbst, Jae Hee Lee, Stefan Wermter
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE181374
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.02178

Browse the full library · About the Tower of Babel