Selective Sinkhorn Routing for Improved Sparse Mixture of Experts
Duc Anh Nguyen, Huu Binh Ta, Nhuan Le Duc, Tan M. Nguyen, Toan Tran · 2025
Read Selective Sinkhorn Routing for Improved Sparse Mixture of Experts by Duc Anh Nguyen, Huu Binh Ta, Nhuan Le Duc, Tan M. Nguyen, Toan Tran (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08972.txt.
Introduction
Duc Anh Nguyen and four co-authors rethink how sparse mixture-of-experts models route tokens to experts. These models grow capacity cheaply by activating only a few experts per token. But keeping the load balanced usually takes extra loss terms or noisy trainable gates. Those complicate training and can misalign the objective.
Treating token-to-expert assignment as an optimal transport problem changes the picture. With balance constraints built into the transport itself, gating scores fall directly out of the transport map, and even minimal transport-based routing already improves the models with no auxiliary losses. The full method, Selective Sinkhorn Routing, keeps that efficiency while preserving flexible expert choice. Language modeling and image classification experiments show gains in training efficiency, accuracy, and robustness when inputs are corrupted.
Catalog record
| CREATOR | Duc Anh Nguyen, Huu Binh Ta, Nhuan Le Duc, Tan M. Nguyen, Toan Tran |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-08972
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Selective Sinkhorn Routing for Improved Sparse Mixture of Experts Duc Anh Nguyen, Huu Binh Ta, Nhuan Le Duc, Tan M. Nguyen, Toan Tran Abstract Sparse Mixture-of-Experts (SMoE) has gained prominence as a scalable and computationally efficient architecture, enabling significant growth in model capacity without incurring additional inference costs. However, existing SMoE models often rely on auxiliary losses (e.g., z-loss, load balancing) and additional trainable parameters (e.g., noisy gating) to encourage expert diversity, leading to objective misalignment and increased model complexity. Moreover, existing Sinkhorn-based methods suffer from significant training overhead due to their heavy reliance on the computationally expensive Sinkhorn algorithm. In this work, we formulate token-to-expert assignment as an optimal transport problem, incorporating constraints to ensure balanced expert utilization. We demonstrate that introducing a minimal degree of optimal transport-based routing enhances SMoE performance without requiring auxiliary balancing losses. Unlike previous methods, our approach derives gating scores directly from the transport map, enabling more effective token-to-expert balancing, supported by both theoretical analysis and empirical results. Building on these insights, we propose Selective Sinkhorn Routing (SSR), a routing mechanism that replaces auxiliary loss with lightweight Sinkhorn-based routing. SSR promotes balanced token assignments while preserving flexibility in expert selection. Across both language modeling and image classification tasks, SSR achieves faster training, higher accuracy, and greater robustness to input corruption. 1 Introduction Large Language Models (LLMs) have rapidly advanced and have been applied in a wide range of domains: natural language processing ( 25 ; 1 ; 21 ) , computer vision ( 7 ; 17 ; 23 ) , and multimodal learning ( 14 ; 22 ) . While scaling up model size by increasing parameters enhances performance, it also leads to higher computational costs and slower inference. To address this problem, Sparse Mixture of Experts (SMoE) ( 24 ; 13 ; 9 ) offers an efficient alternative by dividing the model into specialized sub-networks, or experts, and routing each token to only a few of them. Vanilla SMoE models typically use Softmax-based routing, where each token is sent to the top- k experts based on gating scores. However, this approach often causes routing collapse, where only a few experts are frequently used while others are underutilized ( 3 ) . To address this, prior work has proposed auxiliary losses to encourage balanced expert usage ( 9 ; 13 ) , though these can introduce training instability ( 28 ) . As an alternative, recent methods reformulate routing as an optimal transport (OT) problem, using Sinkhorn-based approaches to achieve a more balanced expert assignment without relying on auxiliary losses ( 16 ) . This promotes stable training, but may limit flexibility, as the gating matrix is not directly optimized through gradients ( 15 ) . In contrast to previous Sinkhorn-based approaches ( 12 ; 4 ; 15 ) , which use the transport map only for choosing the top-k experts, we also use the values of the transport map to assign weights to each chosen expert. Accordingly, routing weights can be derived from the transport map values rather than the gating score matrix (as in existing approaches), harnessing Sinkhorn’s inherent ability to promote balanced expert utilization in SMoE. This design leads to improved expert balancing, as supported by both theoretical analysis and empirical evidence. Building on this insight, we introduce Selective Sinkhorn Routing (SSR), a novel lightweight routing mechanism that replaces auxiliary losses with minimal Sinkhorn-based optimization. Remarkably, applying SSR to just 0.1%–1% of training steps per epoch is sufficient to achieve faster convergence, higher accuracy, and greater robustness to input corruption. We also show that Sinkhorn routing and the noise addition trick improve model performance during training, but have a negative impact at inference time. This work makes the following key contributions: • Propose a new routing framework for SMoE models that combines entropy-regularized OT and stochastic noise injection. This approach promotes balanced expert utilization and enhances training stability. • Provide theoretical results showing that both Sinkhorn-based routing and noise injection are beneficial during training by encouraging exploration and expert balancing, while they must be avoided at inference for consistency and determinism. • Conduct extensive experiments on language modelling and vision tasks, demonstrating that our approach has superior performance compared to existing approaches. 2 Related Work Sparse Mixture of Experts Sparse Mixture-of-Experts (SMoE) ( 24 ; 8 ; 9 ) has emerged as a fundamental backbone for deep learning architectures due to its ability to enhance model capacity while maintaining computational efficiency. Unlike densely activated models, it achieves improved performance in various tasks without incurring excessive computational demands ( 13 ; 27 ) . By activating only a subset of the available experts for each input, SMoE models can scale up the number of parameters significantly without increasing the number of FLOPs per example. This sparsity enables the training of large-scale models that remain cost-effective during inference, making SMoE a favorable architecture for applications such as language modeling and machine translation. Routing in Sparse Mixture of Experts The routing mechanism in SMoE architectures determines how tokens are assigned to experts. A common approach is Softmax-based routing, where a gating network followed by a softmax function computes a distribution over experts for each token ( 24 ) . To ensure that the tokens are evenly distributed among experts, load balancing loss ( 24 ; 13 ; 9 ) or noise terms ( 24 ) are commonly used. Although this improves the load distribution, it introduces additional hyperparameters and training complexity. An alternative approach is Sinkhorn-based routing, which optimizes token-to-expert assignment using the Sinkhorn algorithm ( 4 ; 2 ) . For numerical stability, ST-MoE ( 28 ) introduces an additional z -loss, which penalizes excessively large logits in the gating network. 3 Preliminaries This section provides the necessary foundation on SMoE models, followed by the fundamental background of Sinkhorn-Knopp Algorithm. 3.1 Sparse Mixture of Experts A SMoE model is typically composed of multiple MoE blocks, each of which contains a set of experts. The experts within each block are responsible for processing different aspects of the input, and their outputs are combined to form the block’s final output. Let \mathbf{X}\in\mathbb{R}^{m\times d} be the matrix of m input token embeddings, where each token is represented by a d -dimensional vector and \mathbf{W}_{g}\in\mathbb{R}^{n\times d} be the gating weight matrix for n experts. The gating score matrix \mathbf{S}\in\mathbb{R}^{m\times n} is computed as: \mathbf{S}=\mathbf{X}\mathbf{W}_{g}^{\top} . Each entry s_{i,j} in \mathbf{S} represents the compatibility score between token i and expert j , showing how suitable expert j is for processing token i . Higher scores indicate a stronger preference for routing a token to an expert. For each token i\in\{1,\dots,m\} , we select the top- k experts with the highest scores, denoted by the index set \mathcal{T}_{i}\subseteq\{1,\dots,n\} with |\mathcal{T}_{i}|=k . The routing weight for token i to expert j is then: w_{i,j}=\begin{cases}\displaystyle\frac{\exp(s_{i,j})}{\sum_{j^{\prime}\in\mathcal{T}_{i}}\exp(s_{i,j'})}&,\text{ if }j\in\mathcal{T}_{i},\\ \qquad\quad 0&,\text{ otherwise.}\end{cases} (1) Each expert j is presented by a feedforward network f_{j}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d} . The output of the token i is aggregated as: \mathbf{y}_{i}=\sum_{j\in\mathcal{T}_{i}}w_{i,j}\cdot f_{j}(\mathbf{x}_{i}). (2) 3.2 Sinkhorn-Knopp Algorithm Given two probability vectors r\in\mathbb{R}^{m},s\in\mathbb{R}^{n} and a cost matrix \mathbf{C}\in\mathbb{R}^{m\times n} , the entropy-regularized optimal transport (OT) problem (with entropic regularization parameter \xi>0 ) seeks a transport plan \boldsymbol{\hat{\Pi}}\in\mathbb{R}^{m\times n} that minimizes: \boldsymbol{\hat{\Pi}}=\underset{\boldsymbol{\Pi}\in\mathbb{R}^{m\times n}}{\arg\min}\langle\boldsymbol{\Pi},\mathbf{C}\rangle+\xi\langle\boldsymbol{\Pi},\log\boldsymbol{\Pi}\rangle,