Cross-Benchmark Generalization in Long-Horizon Agents
Sushant Mehta Logan Ritchie Liudas Panavas Edwin Chen Surge AI · 2026
Read Cross-Benchmark Generalization in Long-Horizon Agents by Sushant Mehta Logan Ritchie Liudas Panavas Edwin Chen Surge AI (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2608-00181.txt.
Description
Mehta and co-authors ask whether long-horizon agent training builds transferable skill. In self-contained RL settings, a policy can exploit local quirks. Tool schemas, grader parsing, and task templates can carry rewards. An in-distribution holdout shares those same quirks. Cross-benchmark transfer is the sharper place to look, they argue.
They post-train an open mixture-of-experts model on 363 MCP tasks. Training ran in two stages: supervised fine-tuning, then RL. No external benchmark task or score entered training. The trained model beat its base on five external tests at greedy pass-at-one. Gains reached 9.6 points on Toolathlon. Both software benchmarks improved, though training held no software tasks. Paired trajectory analysis found four recurring behavioural shifts behind the transfer.
Catalog record
| CREATOR | Sushant Mehta Logan Ritchie Liudas Panavas Edwin Chen Surge AI |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2608.00181