CATArena: Evaluation of LLM Agents Through Iterative Tournament Competitions
Lingyue Fu, Xin Ding, Yaoming Zhu, Shao Zhang, Lin Qiu, Weiwen Liu, Weinan Zhang, Xuezhi Cao, Xunliang Cai, Jiaxin Ding, Yong Yu · 2025
Read CATArena: Evaluation of LLM Agents Through Iterative Tournament Competitions by Lingyue Fu, Xin Ding, Yaoming Zhu, Shao Zhang, Lin Qiu, Weiwen Liu, Weinan Zhang, Xuezhi Cao, Xunliang Cai, Jiaxin Ding, Yong Yu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-26852.txt.
Introduction
Lingyue Fu, with eleven co-authors, builds a tournament that tests whether code agents improve over time. Most evaluations of language model code agents grade one-shot code generation. Real development is iterative, so the team introduces CATArena, where agents refine their code across multi-turn tournaments through self-reflection and peer learning from execution feedback.
Fu and the team score agents with a dual metric that separates static generation skill from evolutionary potential. Their experiments show the two are not strictly correlated: a strong starter is not always the best improver. Agents also struggled to use self-reflection and peer learning together for gains. The framework proved extensible and resistant to variance across tasks. The authors offer it as a continuing standard for the evolutionary side of code agents.
Catalog record
| CREATOR | Lingyue Fu, Xin Ding, Yaoming Zhu, Shao Zhang, Lin Qiu, Weiwen Liu, Weinan Zhang, Xuezhi Cao, Xunliang Cai, Jiaxin Ding, Yong Yu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-26852
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CATArena: Evaluation of LLM Agents Through Iterative Tournament Competitions Lingyue Fu, Xin Ding, Yaoming Zhu, Shao Zhang, Lin Qiu, Weiwen Liu, Weinan Zhang, Xuezhi Cao, Xunliang Cai, Jiaxin Ding, Yong Yu Abstract Large Language Model (LLM) agents have evolved from basic text generation to autonomously completing complex tasks through interaction with external tools. However, current benchmarks mainly assess end-to-end performance in fixed scenarios, restricting evaluation to specific skills and suffering from score saturation and growing dependence on expert annotation as agent capabilities improve. In this work, we emphasize the importance of learning ability, including both self-improvement and peer-learning, as a core driver for agent evolution toward human-level intelligence. We propose an iterative, competitive peer-learning framework, which allows agents to refine and optimize their strategies through repeated interactions and feedback, thereby systematically evaluating their learning capabilities. To address the score saturation issue in current benchmarks, we introduce CATArena, a tournament-style evaluation platform featuring four diverse board and card games with open-ended scoring. By providing tasks without explicit upper score limits, CATArena enables continuous and dynamic evaluation of rapidly advancing agent capabilities. Experimental results and analyses involving both minimal and commercial code agents demonstrate that CATArena provides reliable, stable, and scalable benchmarking for core agent abilities, particularly learning ability and strategy coding. 1 1 footnotetext: \dagger Equal contribution. Work done while as Meituan Interns. 1 Introduction With the rapid evolution of agents powered by large language models (LLMs), their capabilities have far surpassed simple text generation. By actively invoking external tools, LLM agents have significantly expanded the boundaries of artificial general intelligence (AGI). These agents are now able to autonomously complete complex, multi-step tasks that are previously considered beyond their reach, such as developing software project ( Manish 2024 ; Hu et al. 2025b ) , intelligently performing strategic planning ( Belle et al. 2025 ) , and learning user preference ( Gao et al. 2024 ) . Existing benchmarks mainly focus on end-to-end performance in specific tasks, such as code generation ( Yang et al. 2024 ) , AI research ( Nathani et al. 2025 ) , and GUI automation ( Wang et al. 2024 ) . These benchmarks provide detailed observations and analyses of LLM agents’ abilities within particular scenarios and have driven significant progress in the field. However, there are important limitations to these approaches. First, the scores obtained in these end-to-end benchmarks only reflect performance on specific tasks , whereas an agent’s overall capability is composed of multiple fundamental skills working together. Second, the absolute scores in these benchmarks, which are typically based on objective correctness, have an upper bound . As agents become increasingly powerful, maintaining and updating these benchmarks requires additional expert-level annotation, and the level of required expertise continues to rise. In light of these challenges, there is an urgent need for a quantifiable and continuously evolving benchmark that systematically measures and analyzes the fundamental sub-abilities of agents. Previous research has shown that self-learning is an essential ability for agents to achieve human-level intelligence ( Gao et al. 2025 ; Zhu et al. 2025 ) . Beyond self-learning, agents, similar to humans, also engage in peer learning, which enables collective evolution through interactions and shared experiences ( Liu et al. 2024 ) . During this evolutionary process, agents receive feedback from their environment and continually improve themselves. This capacity for learning and adaptation is indispensable for LLM agents, as it prepares them for ongoing evolution and more complex challenges. To systematically evaluate this crucial ability, we propose an iterative peer-learning-based competitive framework for LLM agents. In each iteration, agents are required to revise and update their strategies based on the outcomes and policies observed in previous rounds of competition. After every update, the agent policy codes are executed and competed against each other, generating dynamic performance rankings. Through this peer-learning architecture, we gain valuable insights into the learning abilities of LLM agents. Building on this peer-learning framework, we introduce CATArena ( C ode A gent T ournament Arena ) 1 1 1 Code of CATArena is available in https://github.com/AGI-Eval-Official/CATArena. , which utilizes four open-ended, rankable games. These games, including both board games and card games, provide LLM agents with a peer-learning environment and unlimited upper bound for improvement. They enable agents to continually improve and compete, ensuring that the evaluation framework remains challenging as agent capabilities grow. Furthermore, our competitive arena is inherently extensible and can be readily adapted to other types of open-ended, rankable tasks, facilitating the assessment of core agent abilities in new domains. As agent capabilities continue to advance, CATArena can evolve by incorporating tasks with greater complexity and discrimination, thereby supporting ongoing evaluation without the need for expert-level human annotation. In our experiments, we conduct comparative performance evaluations and data analysis conducted on our self-developed minimal code agent and state-of-the-art commercial code agents. CATArena consistently provides stable and reliable benchmarks for assessing both agent capabilities and the agentic potential of the underlying LLMs. Within the peer-learning framework, we design general scoring metrics to systematically assess the fundamental abilities of participating agents, including their learning ability. Our experiments demonstrate that the strategy coding tasks applied in CATArena are fundamentally different from traditional LLM reasoning tasks. This represents a novel evaluation dimension that has not been addressed in previous work. Additionally, we analyze characteristics of CATArena, demonstrating its reliability and extensibility as a benchmarking platform. In summary, our contributions are as follows: • Iterative Peer-learning-based Competitive Framework : We propose a novel framework that leverages iterative peer-learning and competition to evaluate the learning abilities of LLM agents. Agents continuously revise their strategies based on feedback and outcomes from previous rounds, aligning agent evolution with human evolution. • CATArena Benchmark : We introduce CATArena, a tournament-style benchmark for evaluating the basic capabilities of LLM agents using a diverse set of open-ended games, including board and card games. CATArena provides an unlimited upper bound for agent improvement and supports extensible evaluation across diverse, open-ended tasks. • Comprehensive Agent Evaluation : We design general and systematic evaluation matrices and conduct comparative experiments and analyses between our minimal code agent and state-of-the-art commercial agents, demonstrating the reliability, stability, and extensibility of the CATArena. 2 Related Work Learning Ability. Learning ability is crucial for LLM agents, as it enables continual adaptation and improvement in dynamic and complex environments. Recent studies have shown that self-learning methods, such as self-refinement ( Madaan et al. 2023 ; Shinn et al. 2023 ) , allow models to enhance their outputs through iterative feedback, while environmental feedback further supports continual learning ( You et al. 2024 ) . In addition to self-learning, peer-learning has also been increasingly recognized, with approaches encouraging agents to learn from others’ reasoning processes and shared experiences ( Liang et al. 2024 ; Luo et al. 2025 ) . These diverse learning mechanisms have led to notable advances in tasks such as code generation, complex reasoning, and collaborative problem-solving. In the context of LLM-driven agents, learning ability represents a critical capability that supports effective adaptation and enables agents to tackle increasingly complex tasks and evolving challenges ( Zhu et al. 2025 ; Gao et al. 2025 ) . Despite the progress, systematic evaluation of how agents learn from each