Mixture-of-Experts under Finite-Rate Gating: Communication–Generalization Trade-offs

Ali Khalesi, Mohammad Reza Deylam Salehi · 2026

Read Mixture-of-Experts under Finite-Rate Gating: Communication–Generalization Trade-offs by Ali Khalesi, Mohammad Reza Deylam Salehi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-15091.txt.

Description

Khalesi and Salehi study mixture-of-experts models through the lens of communication theory. Such models split prediction across specialized expert sub-networks chosen by a gating step. This letter treats the gate as a stochastic channel working under a finite information rate, asking what limited gating bandwidth costs in learning.

Within an information-theoretic learning framework, the authors specialize a mutual-information generalization bound and develop a rate-distortion account of finite-rate gating. The analysis yields capacity-aware limits for communication-constrained expert systems. Numerical tests on synthetic multi-expert models confirm the predicted trade-offs among gating rate, expressivity, and generalization, tying how much the gate can say to how well the system learns.

Catalog record

CREATORAli Khalesi, Mohammad Reza Deylam Salehi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-15091

Browse the full library · About the Tower of Babel