KALAVAI: Predicting When Independent Specialist Fusion Works A Quantitative Model for Post-Hoc Cooperative LLM Training
Ramchand Kumaresan · 2026
Read KALAVAI: Predicting When Independent Specialist Fusion Works A Quantitative Model for Post-Hoc Cooperative LLM Training by Ramchand Kumaresan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22755.txt.
Introduction
Kumaresan studies when fusing independent specialist models into one model pays off. Contributors fine-tune copies of a shared checkpoint, then a light router merges them. The gain follows a simple rule based on how far the specialists diverge. Below about three percent divergence, gains near zero. At mid-size scales, gains run near seven percent over the best specialist. A cross-lingual fusion gains about twenty-two percent. Shared starting weights and learned routing prove essential. Plain averaging hurts performance, Kumaresan reports.
Catalog record
| CREATOR | Ramchand Kumaresan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 105418 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.22755
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
KALAVAI: Predicting When Independent Specialist Fusion Works A Quantitative Model for Post-Hoc Cooperative LLM Training Ramchand Kumaresan Abstract Independently trained domain specialists can be fused post-hoc into a single model that outperforms any individual specialist, and the gain is predictable before training: gain ≈ 0.82 × divergence − 2.72 \text{gain}\approx 0.82\times\text{divergence}-2.72 ( R 2 = 0.856 R^{2}=0.856 , n = 6 n=6 , spanning 3–26% divergence). A cooperative whose specialists diverge 15% from the shared base checkpoint yields approximately + 7.5 % +7.5\% improvement over the best individual specialist. Below ≈ \approx 3.3% divergence the formula predicts near-zero gain; practitioners can verify specialist divergence before committing training resources. In the Kalavai 1 1 1 KALAVAI ( kalaVAI ) is the ISO 15919 romanisation of the Tamil word for “fusion” or “mixing”. protocol, contributors each fine-tune a copy of a shared base checkpoint on their own data without communication, then submit checkpoints for lightweight MoE routing (500 gradient steps on mixed data). The key property is privacy-preserving distributed training: each contributor’s data never leaves their environment, yet the fused model achieves simultaneous specialist-level quality on every domain ( + 7.72 % +7.72\% vs. best specialist) that centralised training cannot replicate without pooling all data. Fusion gains are consistent across scales: + 7.70 % +7.70\% at 410M (3-seed mean; seed 42: + 7.72 % +7.72\% , ± \pm 0.02pp), + 7.49 % +7.49\% at 1B ( ± \pm 0.01%, 3 seeds), + 6.53 % +6.53\% at 6.9B. The fused model achieves oracle-optimal routing: the learned router matches the domain oracle with gap < 10 − 5 <10^{-5} nats. Phase 2 extends the protocol to high-divergence settings: cross-lingual fusion (Tamil/Yoruba/Welsh/Code) achieves + 21.76 % +21.76\% , with Yoruba perplexity falling 41.9 → 7.7 41.9\to 7.7 . A 20-contributor federation (10 languages + 10 domains) achieves + 16.71 % +16.71\% ( ± \pm 0.07pp, 3 seeds) vs. best specialist. Three structural requirements bound the protocol. Shared initialisation is necessary: checkpoints diverging by as little as one training stage degrade routing clarity. Frozen layers are optional below 10,000 specialist training steps—peak gain + 17.7 % +17.7\% at 5,000 steps without freezing—and recommended beyond that threshold ( + 2.4 +2.4 pp advantage at 20,000 steps). Routing must be learned : uniform equal-weight averaging degrades by − 1.2 % -1.2\% vs. best specialist; a trained linear or MLP router achieves oracle-optimal assignment ( < 10 − 5 <10^{-5} nats gap from domain-level oracle). Hard and soft routing are equivalent once gates are trained. 1 Introduction Training competitive language models requires centralised compute at a scale most researchers and institutions cannot access. A single 70B-parameter training run requires hundreds of A100 GPUs operating synchronously for weeks. This creates a structural barrier: the organisations that can train frontier models are those that can pay for frontier compute, and the rest must fine-tune what is already available. The core insight. A different path exists. If multiple contributors each train a specialist copy of a shared base checkpoint on their own domain, and if those specialists are subsequently fused via a learned router, the resulting model captures complementary knowledge that no single contributor could build alone. The training step requires no communication: contributors work independently, asynchronously, on their own hardware, using their own data. The only coordination is the shared starting point. The fused model requires all N specialists to run at inference, increasing inference compute by a factor of N relative to any individual specialist. This is a training-time democratisation: inference cost scales with N specialists, but so does the knowledge that no single contributor could acquire alone. This observation is not entirely new—the branch-train-mix (BTX) paradigm ( 9 ) demonstrates that MoE fusion of independently trained models is feasible. What is missing from the literature is an empirical characterisation of the conditions under which independent specialist fusion succeeds or fails : when shared initialisation alone is sufficient, when frozen layers become necessary, and what routing architecture drives improvement. The protocol. Kalavai 2 2 2 Code and all experiment scripts: https://github.com/mechramc/Kalavai operationalises cooperative LLM training as a four-step protocol: (1) a coordinator distributes a shared base checkpoint; (2) each contributor fine-tunes independently on their domain for a fixed number of steps; (3) contributors submit their checkpoints; (4) a lightweight router is trained on a small mixed-domain dataset and used for inference. Contributors never share gradients, intermediate activations, or data. The only shared artefact is the initial checkpoint. Key results. We demonstrate: 1. Predictive divergence–gain relationship. Fusion gain scales linearly with specialist divergence: \text{gain}=-2.72+0.82\times\text{divergence} ( R^{2}=0.856 , slope 95% CI [0.35, 1.28]), validated across six conditions from Qwen-1.5B (3.16% divergence, +1.06% gain) to cross-lingual (25.65%, +21.76%). Linear fit is substantially better than log-linear ( R^{2}=0.662 ). The cross-lingual condition exceeds the linear prediction by +3.6pp, consistent with base-model incompetence creating outsized gains. Practitioners can estimate cooperative value from specialist divergence alone before committing to training. 2. Oracle-optimal routing. The learned MoE router matches the domain-level oracle (optimal static assignment of each domain to its best specialist) with a gap of <10^{-5} nats at 410M and 6.9B—effectively zero. The cooperative achieves specialist-level quality on every domain simultaneously, something monolithic training cannot: the monolithic model underperforms the MoE on code ( -4.34\% ) and science ( -3.12\% ), despite seeing all domain data during training. 3. Consistent improvement at scale. Post-hoc MoE fusion beats the best individual specialist by +7.70% at 410M (3-seed mean, \pm 0.02%), +7.49% at 1B (3 seeds, \pm 0.01%), and +6.53% at 6.9B ( \pm 0.024%, 3 seeds) on equal-weight per-domain evaluation. 3 3 3 All results use the per-domain equal-weight protocol implemented in kalavai_eval_utils.py ; see Appendix S . Routing is near-deterministic ( > 99.9%) at all scales. Improvement is stable across Pythia-410M training maturities from 3.5% to 100% of training (+7.0%–+8.8%). 4. Training duration crossover. Without frozen layers, fusion improvement peaks at 5,000 specialist training steps (+17.7%) then degrades to +14.7% at 20,000 steps. With four frozen layers, improvement degrades more slowly (+17.0% at 20,000 steps). The crossover occurs at approximately 10,000 steps, above which frozen layers are recommended. Even 50 training steps produce +4.0% gain—meaningful improvement achievable in under two minutes on a consumer GPU. 5. Routing must be learned. Uniform equal-weight averaging (no routing training) degrades by -1.19\% vs. best specialist; a trained linear router achieves +7.70\% . Hard routing (argmax of learned gates) matches soft routing ( +7.72\% both)—specialist participation drives improvement, not weighting precision. The learned router achieves oracle-optimal domain assignment: gap {<}10^{-5} nats at 410M and 6.9B. 6. Per-domain advantage over monolithic. On aggregate equal-weight loss, MoE and monolithic achieve near-parity ( +0.47\% at 410M). The distinction is per-domain: the MoE matches the best specialist on every domain simultaneously, while the monolithic model sacrifices code ( -4.34\% vs. MoE) and science ( -3.12\% ) to improve on fiction. The cooperative advantage is simultaneous per-domain quality, not lower aggregate loss. This per-domain quality is not achievable by centralised monolithic training, which sacrifices specialist performance on underrepresented domains to minimise aggregate loss. 7. Cross-lingual cooperative training works. Tamil/Yoruba/Welsh/Code specialists fused on Pythia-410M: Yoruba perplexity 41.9 \to 7.7 (5.4 \times ), Welsh 102.7 \to 22.1 (4.6 \times ). Contributors speaking different languages can collectively build