SAT: Sequential Agent Tuning for Coordinator - Free Plug - and - Play Multi - LLM Training with Monotonic Improvement Guarantees Conference: Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026); May 25 – 29, 2026; Paphos, Cyprus
Yi Xie, Yangyang Xu, Yi Fan, Bo Liu · 2026
Read SAT: Sequential Agent Tuning for Coordinator - Free Plug - and - Play Multi - LLM Training with Monotonic Improvement Guarantees Conference: Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026); May 25 – 29, 2026; Paphos, Cyprus by Yi Xie, Yangyang Xu, Yi Fan, Bo Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05216.txt.
Introduction
Xie and three colleagues train teams of small language models without a central coordinator. Big models perform well but cost too much to deploy, so recent work fields teams of smaller models that together rival one giant. Jointly updating several agents, however, creates compounding distribution shifts that destabilize training. SAT, or Sequential Agent Tuning, treats the team as a factorized policy and updates agents one block at a time. An advantage estimator aware of sequence, plus per-agent trust regions, isolate drift.
Theory gives two guarantees: monotonic improvement, so training stays stable, and plug-and-play invariance, so swapping in a stronger agent improves the bound without retraining the rest. A team of three four-billion-parameter agents beat the much larger Qwen3-32B on AIME math benchmarks by nearly four percent on average. Swapping in two eight-billion-parameter agents lifted the composite score by over ten percent.
Catalog record
| CREATOR | Yi Xie, Yangyang Xu, Yi Fan, Bo Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.05216
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SAT: Sequential Agent Tuning for Coordinator - Free Plug - and - Play Multi - LLM Training with Monotonic Improvement Guarantees Conference: Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026); May 25 – 29, 2026; Paphos, Cyprus Yi Xie, Yangyang Xu, Yi Fan, Bo Liu Abstract . Large language models (LLMs) with a large number of parameters achieve strong performance but are often prohibitively expensive to deploy. Recent work explores using teams of smaller, more efficient LLMs that collectively match or even outperform a single large model. However, jointly updating multiple agents introduces compounding distribution shifts, making coordination and stability during training difficult. We address this by introducing Sequential Agent Tuning (SAT), a coordinator-free training paradigm. SAT represents the team as a factorized policy and employs block-coordinate updates over agents, enabling scalable, decentralized training without a central controller. Specifically, we develop a sequence-aware, on-policy advantage estimator that conditions on the evolving team policy, coupled with per-agent KL trust regions that isolate occupancy drift. Theoretically, this framework provides two critical guarantees. First, it ensures monotonic improvement, stabilizing the training process. Second, it establishes provable plug-and-play invariance: any agent can be upgraded to a stronger model without retraining the rest of the team, with a formal guarantee that the performance bound improves. Empirically, a team of three 4B agents (12B total) trained with SAT surpasses the much larger Qwen3-32B on AIME24/25 benchmarks by 3.9% on average. We validate our plug-and-play theory by swapping in two 8B agents, which boosts the composite score by 10.4%. We provide code and appendix of proof at https://github.com/Yydc/SAT-AAMAS \doi 6701030F Author list: Yi Xie, Yangyang Xu, Yi Fan(the work is not related to their position at Amazon Inc), Bo Liu 1. Introduction Large language models (LLMs) have become effective problem solvers across a wide range of domains ( 15 ; 27 ; 26 ; 33 ; 23 ) . Despite their impressive capabilities, state-of-the-art LLMs demand significant memory, computational, and energy resources, making them unavailable for resource-limited scenarios ( 16 ; 1 ) . This disconnect between potential and deployability motivates a key question: Can teams of small, efficient models collectively achieve or surpass the performance of one well-tuned large model? Recent research has investigated multi-agent LLM systems by employing various coordination strategies, such as role assignment (planner, solver), hierarchical workflow, and ensemble refinement protocols ( 28 ; 9 ; 33 ; 25 ; 11 ; 12 ) . While these methods show promising empirical results, they are constrained by two main limitations. First, rely on predefined role assignment, which enforces interaction dynamics and may constrain the potential of LLMs systems ( 35 ) . Second, existing methods generally lack robust theoretical foundations that guarantee multi-agent LLM systems achieve a strong ability, leaving open questions around why multi-LLMs ( 25 ; 23 ) . We address these gaps through a theoretical analysis of sequential agent training with factorized product policies and per-agent trust regions. We establish three properties under a sequence-aware, on-policy advantage estimator that conditions on intermediate policies. Sequence-agnostic improvement states that stage-wise bounds hold regardless of the agent update sequence chosen before each update within the stage. Plug-and-play invariance ensures that agents can be upgraded via a stage-0 KL projection without retraining others while preserving certificates. Certificate tightening indicates that upgrades can increase surrogate values at fixed radii or achieve comparable gains with smaller radii, thereby reducing the cumulative penalty that scales with the sum of the square roots of the radii. The main challenge, covariate shift arising from sequential updates, is addressed by evaluating the advantages under the current intermediate occupancy and constraining the per-agent per-state KL divergence; we also demonstrate that naive rollout reuse without such conditioning fails to guarantee improvement, whereas our approach yields monotonic bounds. Empirical evaluation across general reasoning, active reasoning and planning (AutoLogi PlanBench ) demonstrates that SAT-trained small-model teams match or exceed large baselines under matched evaluation protocols. These gains align with our order - agnostic stage bounds and certificate tightening under per - agent KL trust regions. We summarize our main contributions as follows: • We present a theoretical framework for multi-agent LLM finetuning with monotonic improvement bounds, sequence-agnostic guarantees, and plug-and-play invariance (Sec. 4 ). • We instantiated the theoretical framework through Sequential Agent Tuning (SAT), a coordinator-free training paradigm utilizing sequence-aware block optimization (Sec. 5 ). • We demonstrate that SAT-trained teams outperform strong baselines on 7 benchmarks, with further improvements achieved through plug-and-play paradigm upgrades (Sec . 6 ). 2. Related Work Multi-agent LLM Systems and Orchestration. A line of work explores orchestrating multiple LLMs via roles, tool use, and multi-turn protocols, often with an explicit judge or controller that assigns subtasks or aggregates opinions ( 34 ) . Ensembling-style procedures improve reliability by aggregating diverse completions before selection ( 26 ) . Structured search over thoughts extends this idea with deliberate branching and pruning at the sequence level ( 33 ) . Planning-inspired search further guides generation with MCTS-style rollouts that critique and refine candidates ( 5 ; 31 ) . In contrast, our method is coordinator-free and employs stage-wise sequential updates with per-agent trust regions and an analysis-driven surrogate, yielding provable joint-stage improvement. Monotonic Policy Improvement and Trust Regions. Conservative Policy Iteration establishes monotonic improvement under carefully controlled policy updates ( 7 ) . Trust Region Policy Optimization enforces KL trust regions to stabilize updates with theoretical guarantees ( 18 ) . Proximal Policy Optimization adopts a clipped surrogate that approximates trust-region behavior in practice ( 20 ) . And some early works applying BCD to RL includes ( 37 ; 29 ) . Our development departs by updating agents sequentially while conditioning analysis on the intermediate product policy, and by providing a joint-stage lower bound where per-agent KL penalties accumulate as \sum_{i=1}^{n}\sqrt{\delta_{i}} , justifying small per-agent trust regions. Off-Policy Advantage Estimation under Distribution Shift. Generalized Advantage Estimation (GAE) trades bias for variance reduction by utilizing multi-step returns and a value baseline ( 19 ; 11 ) . Retrace introduces safe multi-step targets via truncated importance weights for off-policy data ( 14 ; 30 ) . V-trace extends this idea to scalable actor–critic training with robust corrections ( 4 ) . In LLM pipelines, iterative procedures such as self-consistency can shift the data distribution across rounds ( 26 ) . Sequence-level search similarly modifies the sampling distribution as the tree expands and prunes candidates ( 33 ) . Our estimator conditions on the updated intermediate policy and uses clipped multi-step ratios to manage bias, leveraging sequence information gathered within a stage. Dynamic Routing and Plug-and-Play Compute. Cascaded inference routes easier inputs to cheaper models and escalates only when necessary, reducing latency without sacrificing accuracy ( 8 ) . Previous multi-agent frameworks may swap tools or models, but they rarely provide guarantees that persist after replacement. Our plug-and-play analysis shows that monotonic improvement certificates remain invariant under agent replacement, provided that the surrogate and trust-region constraints are respected. 3. Preliminaries Environment and policies. Let \mathcal{M}=(\mathcal{S},\{\mathcal{A}_{i}\}_{i=1}^{n},P,r,\gamma) be a discounted MDP with \gamma\in(0,1) and bounded rewards |r|\leq R_{\max} . The joint action space is the product \mathcal{A}=\mathcal{A}_{1}\times\cdots\times\mathcal{A}_{n} . A team of n execution agents forms a factorized policy over