Does Role Specialization Matter for Explanation Faithfulness in Mixture-of-Experts?

Yeji Kim(✉), Housam Babiker, Mi-Young Kim, Randy Goebel · 2026

Read Does Role Specialization Matter for Explanation Faithfulness in Mixture-of-Experts? by Yeji Kim(✉), Housam Babiker, Mi-Young Kim, Randy Goebel (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29613.txt.

Description

Kim, Babiker, Kim, and Goebel ask whether giving experts distinct roles inside a Mixture-of-Experts model keeps its explanations faithful. Such architectures route work to specialist components, and role-based versions label experts with meanings such as synergy, redundancy, and uniqueness. A labelled role, though, does not guarantee that the model's explanations reflect its real computation.

The team hypothesizes that overlap between experts' internal representations weakens real role separation and degrades attribution-based faithfulness. They add representation-level decorrelation regularization, a training penalty that pushes experts' representations apart. Across several multimodal benchmarks, meaning tests that mix images and text, faithfulness improves on measures of comprehensiveness and sufficiency. Task performance stays intact at the same time. The same trend appears in a standard sparse Mixture-of-Experts baseline without explicit roles. The authors conclude that structural role labels alone are insufficient; separation in representation space does the work.

Catalog record

CREATORYeji Kim(✉), Housam Babiker, Mi-Young Kim, Randy Goebel
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.29613

Browse the full library · About the Tower of Babel