Cross-Benchmark Generalization in Long-Horizon Agents

Sushant Mehta Logan Ritchie Liudas Panavas Edwin Chen Surge AI · 2026

Read Cross-Benchmark Generalization in Long-Horizon Agents by Sushant Mehta Logan Ritchie Liudas Panavas Edwin Chen Surge AI (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2608-00181.txt.

Description

Mehta and co-authors ask whether long-horizon agent training builds transferable skill. In self-contained RL settings, a policy can exploit local quirks. Tool schemas, grader parsing, and task templates can carry rewards. An in-distribution holdout shares those same quirks. Cross-benchmark transfer is the sharper place to look, they argue.

They post-train an open mixture-of-experts model on 363 MCP tasks. Training ran in two stages: supervised fine-tuning, then RL. No external benchmark task or score entered training. The trained model beat its base on five external tests at greedy pass-at-one. Gains reached 9.6 points on Toolathlon. Both software benchmarks improved, though training held no software tasks. Paired trajectory analysis found four recurring behavioural shifts behind the transfer.

Catalog record

CREATORSushant Mehta Logan Ritchie Liudas Panavas Edwin Chen Surge AI
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2608.00181

Browse the full library · About the Tower of Babel