Learning to Hand Off: Provably Convergent Workflow Learning under Interface Constraints
Jiayu Li, Enpei Zhang, Dawei Zhou, Elynn Chen, Yujun Yan · 2026
Read Learning to Hand Off: Provably Convergent Workflow Learning under Interface Constraints by Jiayu Li, Enpei Zhang, Dawei Zhou, Elynn Chen, Yujun Yan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19140.txt.
Introduction
Jiayu Li and four colleagues study workflow learning where agents hand off an artifact, each seeing only part of it plus its state. They formalise this as a semi-Markov decision process with interface constraints and design IC-Q, a Q-learning algorithm whose coordination at each handoff is one scalar. Their result bounds IC-Q error in samples across function-approximation error, interface gap, and a mixing-time residual. Four experiments show IC-Q matches an oracle with no agent seeing joint trajectories, with error sources scaling as predicted.
Catalog record
| CREATOR | Jiayu Li, Enpei Zhang, Dawei Zhou, Elynn Chen, Yujun Yan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43571 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.19140
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning to Hand Off: Provably Convergent Workflow Learning under Interface Constraints Jiayu Li, Enpei Zhang, Dawei Zhou, Elynn Chen, Yujun Yan Abstract We study workflow learning in a setting where specialized agents hand off control through a shared artifact, each agent observes only a local function of that artifact and its own private state, and no centralized learner accesses joint trajectories – the operating regime of multi-agent LLM pipelines that span organizational, vendor, or trust boundaries. We formalize this regime as an interface-constrained semi-Markov decision process (IC-SMDP), whose decision epochs occur at handoff times, and design IC- Q Q , an asynchronous decentralized Q Q -learning algorithm in which cross-agent coordination at every handoff is exactly one scalar. Our main result is a finite-sample bound for neural IC- Q Q that decomposes into three independently controllable error sources: neural function-approximation error, interface representation gap, and a mixing-time residual, under the random option-duration discount. Establishing this bound requires lifting the approximate information state (AIS) framework from single-agent primitive-step MDPs to multi-agent SMDPs and controlling Markovian noise under random duration, neither of which has been done in prior work. To our knowledge this is the first finite-sample guarantee for neural Q Q -learning under decentralized partial observability. Four experiments: a controlled synthetic IC-SMDP that validates the bound term-by-term, multi-LLM mathematical reasoning, multi-agent routing, and multi-agent CPU programming, show that IC- Q Q matches a centralized oracle without any agent observing joint trajectories, with each of the three error sources scaling along its corresponding axis as the bound predicts. 1 Introduction Multi-agent LLM systems coordinate specialized agents such as planners, coders, testers, retrievers, verifiers hong2023metagpt ; huang2023agentcoder ; gottweis2025aiscientist ; yao2023react into workflows whose end-to-end performance is shaped by how the workflow is structured : which agent acts first, when control transfers, and what information passes between them. Two distinct regimes for designing such workflows have emerged. The centralized regime assumes a single designer with global visibility into agent trajectories. Hand-designed workflows hong2023metagpt ; wu2023autogen fix the structure in advance, while learned workflow generators zhuge2024gptswarm ; zhang2024aflow ; hu2024adas ; fan2024workflowllm ; yue2025daao optimize it from joint trajectory data. This regime is well-developed, and, where its assumptions of centralization hold, it works. The sequential decentralized regime is structurally different. Production systems are increasingly built as pipelines of role-specialized agents that transfer one artifact at a time, as in the assembly-line architecture of MetaGPT hong2023metagpt , the programmer-to-tester pipeline of AgentCoder huang2023agentcoder , the turn-based message-passing of AutoGen wu2023autogen . Emerging agent-to-agent protocols anthropic2024mcp ; google2025a2a standardize sequential message exchange as the coordination primitive. When such pipelines span organizational, vendor, or trust boundaries yang2025agentnet , the centralized assumptions fail: no party holds joint trajectories at training time, agents see only the artifact they receive together with their own private state, and chains of thought, scratchpads, and proprietary prompt templates are not exposed across agent boundaries by design, by safety filtering, or by API contract. The resulting decision problem combines partial observability with the cross-environment heterogeneity studied in latent-heterogeneous RL chen2024reinforcement , where each agent’s local view induces effectively a distinct learning problem. Workflow learning in this regime has, to date, been addressed only by hand-designed adaptive rules yang2025agentnet without a decision-theoretic foundation or convergence guarantee. This paper provides such a foundation. We study workflow learning in the sequential decentralized regime under four operating conditions that distinguish it from the centralized setting: (i) sequential handoff-based control – exactly one agent acts at each step and transfers control through a shared artifact; (ii) decentralization in both training and execution – no party accesses joint trajectories at any stage; (iii) interface-limited observation – each agent decides on the basis of the artifact it receives plus its own private state, with no exposure of internal state across agent boundaries; and (iv) finite-sample guarantees – deployments run under bounded API and compute budgets, so a designer needs to know how much sample budget suffices, not merely that convergence eventually obtains. Finite-sample analysis at this granularity has recently driven theoretical progress in adjacent areas, including transfer Q -learning chen2025transfer ; chai2025transfer and high-dimensional sequential decision-making under structured latent heterogeneity chen2025highdim , and motivates an analogous treatment for decentralized agentic workflows. Why existing frameworks do not extend. Four lines of prior theory bear on this regime; each violates at least one of the four conditions above. \bullet Decentralized POMDPs and CTDE. Dec-POMDPs bernstein2002decpomdp ; nair2005networked ; oliehoek2016decpomdp ; oliehoek2008exploiting model cooperative multi-agent control under local observations but assume concurrent action with a joint reward, violating sequentiality. Centralized training with decentralized execution lowe2017maddpg ; rashid2020qmix ; foerster2018coma ; sunehag2018vdn ; iqbal2019actorattentioncritic relaxes execution but requires joint-trajectory access at training, violating decentralization. Related transfer-RL formulations that share information across heterogeneous tasks chen2026data ; chai2026optimistic likewise presume a coordinator who observes either joint trajectories or task-level structure unavailable across vendor boundaries. \bullet The options framework. Options sutton1999options ; bacon2017optioncritic ; bradtke1994smdp ; precup2000options provide the natural temporal-extension primitive but assume a single meta-controller observing the full state at each decision epoch, violating interface-limited observation. Recent work on prior-aligned meta-RL with finite-horizon guarantees zhou2025prior inherits the same full-observation assumption at the meta-controller level. \bullet Approximate information states. The AIS framework subramanian2022ais ; kara2022finite ; sinha2024agentstate ; sinha2024periodic treats single-agent partial observability in primitive-step MDPs. Lifting it to decentralized AIS observations across N agents composing through handoffs requires controlling the AIS error under the random discount \gamma^{\tau_{k+1}} and over a disjoint observation space – neither has been done. The common-information extension of kao2022common addresses concurrent Dec-POMDPs through a fictitious coordinator and is not applicable to sequential handoff control. Closely related are non-stationary RL settings where the environment itself shifts across episodes chai2025deep , for which transfer guarantees require additional structural assumptions that the IC-SMDP does not impose. \bullet Multi-agent LLM orchestration. Recent systems yang2025agentnet ; zhuge2024gptswarm ; zhang2024aflow ; hu2024adas route tasks through learned or hand-designed DAGs of LLM agents. AgentNet yang2025agentnet is closest in spirit but operates via hand-designed adaptive rules (moving averages over edge weights, retrieval-based heuristics, capability-vector updates) with no Bellman recursion and no convergence guarantee, violating finite-sample guarantees. Workflow learning has also been studied for economic and operational decision-making, where finite-sample optimality under cross-market heterogeneity chen2026transfer ; zhang2025transfer has driven recent theoretical advances – none of which apply directly to the handoff-based interface regime we study. A full discussion appears in Appendix . Contributions. We provide the first framework that respects all four conditions and prove finite-sample convergence inside it. \bullet Formal model. We introduce the interface-constrained semi-Markov decision process (IC-SMDP) (§ 2 ) and show that it induces a well-defined SMDP at handoff times to which the AIS framework lifts with quantifiable interface gap (\varepsilon_{\phi},\delta_{\phi}) . \bullet Decentralized algorithm. We design IC- Q (§ 3 ), an asynchronous Q -learning algorithm in which cross-agent coordination at every handoff is a single scalar – minimal communication overhead under the bandwidth and API-call constraints of cross-vendor deployment. \bullet Finite-sample convergence. We prove a finite-sample bound (Theorem 1 , § 4 ) decomposing into three independently controllable error sources – neural approximation, interface representation gap, and mixing-time residual. Three challenges arise that prior analyses do not face: a Bellman contraction under random rather than