Beyond Routing: Characterising Expert Tuning and Representation in Vision Mixture-of-Experts
Gene Tangtartharakul, Katherine R. Storrs · 2026
Read Beyond Routing: Characterising Expert Tuning and Representation in Vision Mixture-of-Experts by Gene Tangtartharakul, Katherine R. Storrs (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20610.txt.
Introduction
Gene Tangtartharakul and Katherine Storrs look past routing to ask what experts in vision mixture-of-experts models actually encode. They train sparsely gated convolutional experts with a contrastive goal on natural images and probe tuning with neuroscience tools. An animate-inanimate split dominates how experts divide labour, from gating through to readout, and stays stable across independent trainings. Routing alone, they show, suggests sparser and more categorical preferences than experts truly hold.
Catalog record
| CREATOR | Gene Tangtartharakul, Katherine R. Storrs |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52013 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.20610
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Routing: Characterising Expert Tuning and Representation in Vision Mixture-of-Experts Gene Tangtartharakul, Katherine R. Storrs Abstract Mixture-of-Experts (MoE) models are often interpreted by analysing which categories are routed to which experts. However, routing alone does not reveal what each expert actually encodes. We train sparsely-gated convolutional MoE models with a contrastive objective on natural images and characterise expert specialisation using tools from visual neuroscience. Extending from gating-level to expert-level analyses, we measure per-expert category separability, and per-expert tuning using the most exciting inputs. Extending from category-level to feature-level explanations, we interpret tuning via semantic dimensions derived from a dataset of human behavioural judgements (THINGS). Finally, we use tuning and representational similarity analysis to assess the stability of expertise-allocation across independent initialisations. We find that an animate–-inanimate distinction dominates expert partitioning, apparent from gating through to expert readout, and is stable across independently trained models. Although routing statistics suggest relatively sparse, categorical preferences, expert analyses reveal broader tuning to continuous visual and semantic dimensions that extend beyond category boundaries. Experts exhibit similar category-separability to one another, despite distinct feature tuning, demonstrating the explanatory benefits of moving beyond category-level analyses. Together, these results show that expert specialisation in vision MoEs extends well beyond category routing and is better understood by probing fine-grained expert-level tuning and representational structure. 1 Introduction Mixture-of-Experts (MoE) has become an increasingly common architecture in deep neural networks across modalities and applications. Rather than relying on a single dense network, MoE models comprise multiple smaller, distinct subnetworks, or experts , together with a gating network that learns to allocate inputs to appropriate subnetworks [ 14 , 24 , 50 , 25 , 39 ] . After training, experts are often found to capture different components of a task. Modern implementations of MoE architectures typically induce sparsity [ 14 , 17 , 48 , 50 , 51 ] , such that only k experts (out of E ) are activated for a given input, where k\ll E . MoE design naturally encourages expert specialisation by partitioning a task into disentangled subspaces that can be associated with different experts, thereby aiding interpretability [ 15 , 60 ] . Understanding what a model has learned and why is crucial for revealing internal representations, diagnosing failures, and identifying targets for alignment and model improvement. MoE architectures provide an explicit inductive bias toward interpretability, as each expert can be inspected in isolation, making it possible to identify which input features or subtasks activate which expert [ 44 , 40 , 56 ] . Lightweight gating also improves accessibility into the model’s internal routing and decision-making processes. Several studies have examined MoE interpretability [ 40 , 11 , 17 , 48 , 56 , 54 , 5 , 6 , 32 , 42 , 61 , 41 ] . In language MoEs, expert specialisation often reflects syntactic structure, semantic structure, or a combination of both [ 2 , 62 , 50 , 42 , 58 ] , with studies commonly reporting specialisation for parts of speech [ 2 , 62 , 50 ] . In vision MoEs, experts are often reported to specialise according to semantic categories (e.g., animals and plants) [ 44 , 48 , 41 ] , object size [ 44 ] , or spatial region [ 17 , 43 ] , depending on the training objective [ 43 , 54 ] . In multimodal MoEs, experts may likewise partition inputs within or across modalities [ 51 ] . Across MoE studies, expert specialisation is almost exclusively probed through gating information. In language MoEs, routing decisions are examined through token-to-expert assignments and usage frequency [ 2 , 62 , 50 ] ; in vision MoEs, through gating weights and expert-by-class heatmaps [ 44 , 48 , 41 ] ; and in multimodal MoEs, through modality-usage charts [ 51 ] . As a result, existing work largely treats routing decisions as a proxy for specialisation rather than directly examining the experts themselves, leaving the representations and feature-tuning profiles experts learn poorly understood. Analogous questions have been studied extensively in visual neuroscience. The occipitotemporal cortex (OTC) contains functionally specialised modules with well-documented selectivity for colour [ 37 ] , shape [ 36 , 16 ] , faces [ 27 , 26 ] , places [ 13 ] , body parts [ 10 ] , and words [ 18 , 33 ] . Although these regions are traditionally described as category-selective, accumulating evidence favours a more distributed account: rather than being tuned to a single category, representations within these regions are broadly sensitive to finer-grained, behaviourally relevant dimensions [ 8 , 12 , 35 , 49 ] . This motivates a broader interpretation of expert specialisation in MoEs, and suggests that the neuroscientific toolkit for characterising distributed representations may offer a useful lens for probing vision MoEs. We return to this point in the next section. In this work, we build a sparsely gated contrastive Mixture-of-Experts convolutional neural network and characterise expert specialisation using tools drawn from visual neuroscience. Our contributions are: 1. Beyond routing. We extend MoE interpretability from gating-level statistics to expert-level tuning and population-level representational geometry, showing that routing alone systematically mischaracterises what experts encode. 2. Beyond categories with continuous perceptual dimensions. We use the THINGS dataset [ 22 ] and its 66 human behaviour-derived visual and semantic dimensions [ 23 ] as an out-of-distribution probe, disentangling expert tuning from the categorical structure of the training set and recovering interpretable, continuous tuning profiles. 3. Representational stability. Across independently trained models ( E\in\{4,8,16\} , 10 seeds each), we identify representational motifs that re-emerge reliably using second-order RSA and silhouette-based cluster selection. Together, these results reframe expert specialisation in vision MoEs as sparse tuning to continuous, behaviourally relevant dimensions rather than to discrete categorical classes. 2 Related Work Interpretability of Vision and Multimodal MoEs. MoE layers have been integrated with various vision backbones, including ViT [ 48 ] , VAE [ 41 ] , and CNN [ 44 ] . Across these works, expert specialisation is predominantly characterised through routing dynamics as a proxy: per-class averages of routing weights or logits [ 57 ] , routing frequency [ 17 , 53 ] , and visualisations of gating patterns via low-dimensional embeddings or per-expert exemplars [ 44 , 41 ] . In multimodal MoEs, one study extended beyond routing to within-expert probing [ 61 ] by adapting the Model Utilisation Index (MUI) to MoE architectures. This measure quantifies the proportion of neurons within each expert activated for task completion. Nevertheless, such a probing technique remains a functional measure of recruitment, as it identifies which neurons are engaged but not what they encode, leaving expert tuning and representational structure unexplored. Functional Analyses of Cortical Visual Areas. In neuroscience, various tools have been developed to characterise the tuning properties of cortical areas across multiple scales. Tuning at the level of individual neurons or neural populations can be probed using techniques such as Most Exciting Inputs (MEIs), which have long been used in animal [ 1 , 19 ] and human neurophysiology, and more recently in deep neural networks [ 55 ] , to identify the stimuli that maximally drive individual neurons or neural populations. MEIs can be selected