Mixture of Thoughts: Learning to Aggregate What Experts Think, Not Just What They Say
Jacob-Fein Ashley, Dhruv Parikh, Rajgopal Kannan, Viktor Prasanna · 2025
Read Mixture of Thoughts: Learning to Aggregate What Experts Think, Not Just What They Say by Jacob-Fein Ashley, Dhruv Parikh, Rajgopal Kannan, Viktor Prasanna (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-21164.txt.
Introduction
Different open language models specialize in mathematics, code, or general reasoning, and combining them usually means picking just one per query. Jacob Fein-Ashley and three co-authors introduce Mixture of Thoughts in a 2025 paper. The method lets expert models collaborate inside their hidden states, not just trade finished text.
For each query, a lightweight router picks the top experts and names one as primary. Interaction layers, placed at uniform depths, project hidden states into a shared space so experts can exchange partial thinking. Earlier approaches routed queries, ran costly multi-turn exchanges, or merged weights, which demands identical architectures. Mixture of Thoughts works across heterogeneous models. The authors argue aggregation should capture what experts think, not only what they say.
Catalog record
| CREATOR | Jacob-Fein Ashley, Dhruv Parikh, Rajgopal Kannan, Viktor Prasanna |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-21164
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mixture of Thoughts: Learning to Aggregate What Experts Think, Not Just What They Say Jacob-Fein Ashley, Dhruv Parikh, Rajgopal Kannan, Viktor Prasanna Abstract Open-source Large Language Models (LLMs) increasingly specialize by domain (e.g., math, code, general reasoning), motivating systems that leverage complementary strengths across models. Prior multi-LLM approaches either (i) route a query to one or a few experts and generate independently, (ii) aggregate outputs from each model via costly multi-turn exchanges, or (iii) fuse weights into a single model—typically requiring architectural homogeneity. We introduce Mixture of Thoughts (MoT), a simple method for latent-level collaboration among heterogeneous experts under a global routing scheme. For each query, a lightweight router selects top- K K experts and designates a primary expert; uniformly placed interaction layers project hidden states into a shared latent space where the primary expert performs cross-attention over its active (selected) peers. Pre-trained experts remain frozen; only the router and the lightweight interaction layers are trained with a novel joint training objective that improves both the expert selection and inter-expert collaboration. Across five in-distribution (ID) and three out-of-distribution (OOD) benchmarks, MoT surpasses the current routing and aggregation-based state-of-the-art, Avengers , by +0.38% and +2.92%, respectively. Further, MoT significantly outperforms the best-performing single model. It achieves this with single-pass inference, runtime comparable to routing baselines, and none of the overheads of iterative aggregation. MoT offers a simple latent-space mechanism for combining heterogeneous LLMs, a practical step toward broader multi-LLM collaboration. Our code is publicly available at https://github.com/jacobfa/mot . Figure 1: Accuracy radar plot across OOD and ID tasks, showing MoT outperforming base models and state-of-the-art routing baselines. Scores are normalized to a 0-100 range for visualization. 1 Introduction Large Language Models (LLMs) have reshaped applications in mathematics, coding, multimodal reasoning, and language understanding llmsurvey . Open-source LLMs (e.g., Mistral mistral7b , LLaMA 3 llama-3 , LLaMA 4 llama-4 , Qwen 3 qwen-3 ) are competitive, yet closed-source systems (GPT-5 gpt-5 , Claude Opus 4.1 claude-opus-4.1 , Grok 4 grok-4 , Gemini 2.5 gemini-2.5 ) still lead on widely used and highly competitive benchmarks, including MMLU mmlu , GSM8K gsm8k , BBH big-bench , HumanEval humaneval , ARC-Challenge ARC-Challenge , GPQA GPQA , MMMU/MMMU-Pro mmmu , mmmu-pro , SWE-Bench Verified swe-bench , swe-bench-verified , and Humanity’s Last Exam humanity-last-exam . Specialized open-source models excel in particular skills—math meta-math , code llm-coding , or commonsense llm-common-sense-reasoning —suggesting complementary expertise that could be combined. Prior multi-LLM efforts do so via: (i) selective routing to one or a few experts routerdc , zooter , embed-llm , routellm , graphrouter , routerbench , universal-routing , llm-blender , model-sat , avengers , deepen , sometimes with ensemble sampling avengers ; (ii) response-level collaboration (mixture-of-agents) moa , sparse-moa , self-moa , symbolic-moe ; and (iii) parameter-space fusion (merging, continual learning, distillation) ties-merging , DARE , model-swarms , lorahub , genome , model-fusion-homogeneity , fusellm . These strategies are coarse-grained and/or compute-heavy, often preserving only narrow skills and weakening generalization survey-multiple-llm . Limitations. Learned routing (e.g., RouterDC routerdc ) maps each query to a single expert, binding performance tightly to the chosen expert without any cross-model interaction. Multi-expert routing (e.g., Avengers avengers , current state of the art) relaxes this by selecting multiple experts, but combines their outputs through ensembling via sampling and voting wang2022self , multi-sample , rather than true collaboration. While response-level collaboration (e.g., MoA moa ) aggregates final outputs (responses) from multiple models, it does so at the cost of multi-turn iterative exchanges between models. Parameter fusion approaches (e.g., TIES-Merging ties-merging , FuseLLM fusellm ) require architectural homogeneity and collapse specialization into a single set of weights, removing per-query adaptivity. Mixture of Thoughts (MoT). All prior paradigms lack fine-grained interaction in the latent space, restricting collaboration to outputs, weight fusion, or naive ensembling. To address this gap, we propose Mixture of Thoughts ( MoT ), a method for latent-level collaboration among heterogeneous experts under a global router. For each query, a lightweight trainable router selects the top- K experts and designates a primary expert. Within each expert, we uniformly place interaction layers composed of lightweight adapters (projections) and a cross-attention module. At these layers, all active experts’ hidden states are projected into a shared latent space and integrated into the primary expert via cross-attention—enabling fine-grained collaboration while keeping each expert’s backbone frozen. A joint training objective optimizes both the router and interaction layers. MoT performs a single forward pass through selected experts, achieving routing-like efficiency with richer representational capacity than single- and multi-model routing, response-level aggregation, and parameter fusion, while supporting fully heterogeneous experts with dynamic, per-query adaptive collaboration (Fig. 2 ). Unlike prior work that aggregates only at the output ( ‘say’ ), MoT enables experts to integrate hidden representations ( ‘thoughts’ ) via interaction layers. Figure 2: Prior paradigms (routing, output aggregation) vs. our MoT schematic. Parameter fusion schematic not shown as it does not support heterogeneous models. Dark green lines are for selected experts. We evaluate MoT on language understanding, code generation, and mathematical reasoning under in-distribution (ID) and out-of-distribution (OOD) settings. MoT attains state-of-the-art performance among open-source multi-LLM methods: it surpasses the strongest single model (Dolphin-2.9-LLaMA-3-8B) by +10.95 % (ID) and +9.03 % (OOD), improves over RouterDC by +3.40 % (ID) and +4.52 % (OOD), and exceeds Avengers by +0.38 % (ID) and +2.92 % (OOD), with comparable wall-clock cost (51.8 vs. 51.3 minutes). MoT is robust to the number of selected experts, expert drop-out, the number of interaction layers (and their placement), and shared latent-space dimensionality. Capability R C F MoT Latent-level collaboration \times \times \times \checkmark Cost-efficient (single-pass) \checkmark \times \checkmark \checkmark Supports heterogeneous experts \checkmark \checkmark \times \checkmark Per-query collaborative adaptation \times \checkmark \times \checkmark Beyond output-level fusion \times \times \checkmark \checkmark Representative works: R (routing): routerdc , zooter , embed-llm , routellm , universal-routing , llm-blender , graphrouter , routerbench , model-sat , avengers , deepen ; C (response-level collaboration): moa , sparse-moa , self-moa , symbolic-moe , avengers ; F (fusion): ties-merging , DARE , genome , model-fusion-homogeneity , model-swarms , lorahub , fusellm ; Contributions. (i) MoT : combines a lightweight global router with latent-level interaction layers, enabling per-query collaboration among heterogeneous, frozen experts. (ii) A joint training objective optimizes the router and interaction layer parameters. (iii) Single-pass multi-expert inference yields lower overhead than response-level collaborative methods (e.g., MoA ) and greater representational capacity than routing-only baselines (e.g., RouterDC , Avengers ). (iv) Consistent ID/OOD gains: +3.40 % / +4.52 % over RouterDC , and +0.38 % / +2.92 % over Avengers , at comparable wall-clock cost. (v) Robust to expert count, drop-out, interaction depth/placement, and shared latent size; preserves specialization while enabling dynamic query-level collaboration. 2 Related Work Routing (Selection-Based). Routing-based integration methods use a learned or heuristic router to select one (or a small subset) of experts per query. RouterDC routerdc trains a parameter-efficient dual-contrastive router; Zooter zooter employs zero-shot prompt scoring; EmbedLLM embed-llm embeds queries and models in a joint space; RouteLLM routellm learns task-specific gates; and GraphRouter graphrouter models skill relationships over a graph. RouterBench routerbench evaluates routers across tasks, and universal-routing proposes task-agnostic, transferable routers. Multi-model variants ( LLMBlender llm-blender , Model-SAT model-sat , Avengers avengers , Deepen deepen ) select