Efficient Agents: Building Effective Agents While Reducing Cost
2025
Read Efficient Agents: Building Effective Agents While Reducing Cost (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-02694.txt.
Introduction
Wang and co-authors ask, in a 2025 study, how much of a language model agent's cost is fat and how much is muscle. Agents ace hard multi-step tasks, but their bills threaten to shut out smaller users. The team ran the first systematic sweep of the efficiency-effectiveness trade-off on the GAIA benchmark.
They varied the model backbone, the framework design, and test-time scaling, pricing each option with a cost-of-pass measure. Their trimmed framework, Efficient Agents, kept 96.7 percent of a leading system's performance while cutting run cost from about forty cents to twenty-three, a 28.4 percent better cost-of-pass. Many agent modules, it turns out, add price faster than skill. The study hands builders a map of where the money actually buys results.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-02694
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Efficient Agents: Building Effective Agents While Reducing Cost Abstract The remarkable capabilities of Large Language Model (LLM)-driven agents have enabled sophisticated systems to tackle complex, multi-step tasks, but their escalating costs threaten scalability and accessibility. This work presents the first systematic study of the efficiency-effectiveness trade-off in modern agent systems, addressing the critical need for cost-effective designs without sacrificing performance. We investigate three key questions: (1) How much complexity do agentic tasks inherently require? (2) When do additional modules yield diminishing returns? (3) How much efficiency can be gained through the design of efficient agent frameworks? Through an empirical analysis on the GAIA benchmark, we evaluate the impact of LLM backbone selection, agent framework designs, and test-time scaling strategies. Using the cost-of-pass metric, we quantify the efficiency-performance trade-off across these dimensions. Our findings inform the development of Efficient Agents , a novel agent framework that has an optimal complexity to task requirements. Efficient Agents retains 96.7% of the performance of OWL, one leading open-source agent framework, while reducing operational costs from $0.398 to $0.228, resulting in a 28.4% improvement in cost-of-pass. Our work provides actionable insights for designing efficient, high-performing agent systems, advancing the accessibility and sustainability of AI-driven solutions. \correspondence Wangchunshu Zhou at ; He Zhu at \checkdata [Code] https://github.com/OPPO-PersonalAI/OAgents 1 Introduction The ever-increasing reasoning and creation capabilities of Large Language Models (LLMs) have opened up a broad prospect for real-world applications. Researchers have developed numerous LLM-driven agent systems [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 ] and created a large number of fascinating products capable of handling complex, multi-step tasks. However, this progress mirrors a familiar trajectory in NLP research: from BERT [ 11 ] to ChatGPT [ 12 ] , researchers consistently prioritize scaling up models to achieve breakthrough capabilities [ 13 , 14 ] , only later turning to optimize efficiency, cost, and environmental impact [ 15 , 16 ] . This pattern has given rise to the critical subfield of efficient NLP, where researchers balance performance with practical constraints like inference latency, energy consumption, and economic viability. We argue that agent research has now reached a similar inflection point. While increasingly sophisticated agent architectures can solve remarkably complex problems, their costs scale prohibitively. Industry deployments reveal this tension starkly: cutting-edge agent products (e.g., DeepResearch [ 17 ] , Manus [ 18 ] ) demonstrate impressive capabilities but suffer from exorbitant operating costs due to explosive LLM call overhead. Some systems require hundreds of API calls per task, rendering them economically unsustainable despite their technical brilliance. This creates a fundamental bottleneck for real-world adoption, limiting both the scalability of applications and the accessibility of AI advancements. Our work presents the first systematic study of the efficiency-effectiveness trade-off in modern agent systems. Through rigorous empirical analysis, we investigate 3 research questions: (1) How much complexity do agentic tasks truly require? (2) When do additional modules yield diminishing returns? (3) How much efficiency can be gained through the design of task-adaptive agent frameworks? By dissecting these relationships across the framework, we provide actionable insights for both researchers and practitioners. We conduct an empirical study on the GAIA benchmark [ 19 ] focusing on the efficiency-performance trade-off of agent systems by evaluating the impact of: (1) the choice of LLM backbones; (2) the designs of agent frameworks including planning, tool using, and memory; (3) test-time scaling and ensemble strategies. We adopt the cost-of-pass [ 20 ] metric to compare and analyze the impact of different design choices on the efficiency-performance trade-off of LLM-based agents, as illustrated in Figure 1 . Based on the insight obtained in the aforementioned empirical study, we introduce Efficient Agents , an agent framework optimized for achieving the best efficiency-performance trade-off that achieves a new state-of-the-art on the cost-of-pass metric on the GAIA benchmark. Specifically, 96.7% of the performance of OWL [ 21 ] , an open-source agent framework that achieves great performance on the GAIA benchmark, while reducing the cost from $0.398 to $0.228, leading to a relative improvement of 28.4% in terms of cost-of-pass. Our contribution could be summarized as follows: • We thoroughly analyze and summarize the factors that cause significant economic overhead in a generic LLM-based agent system. • We propose Efficient Agents , an efficient agent system where each component is selected based on prior analytical results, optimizing for efficiency while maintaining high effectiveness. Experimental results demonstrate that Efficient Agents achieves 96.7% of the performance of OWL while reducing costs by 28.4%. Figure 1 : Evaluation of effectiveness and efficiency in agent system components. We adopt cost-of-pass as the metric to evaluate. We develop Efficient Agents that optimizes cost while maintaining accuracy. 2 Preliminaries 2.1 Setup Many factors can influence the effectiveness and efficiency of an agent system [ 22 ] . In this paper, we aim to conduct a comprehensive analysis from the perspective of agent systems. These factors encompass not only the backbone LLM itself but also the agent framework built around it, including planning mechanisms [ 23 ] , tool usage [ 24 ] , and memory module [ 25 ] . In addition, test-time scaling strategies [ 26 ] are also considered. To this end, we evaluate these components on GAIA [ 19 ] , a popular and challenging agent benchmark. This benchmark typically requires agents to perform complex reasoning to solve problems. By leveraging the benchmark, we can effectively assess the impact of individual components on overall performance and efficiency. Additionally, we compare several distinct agent frameworks to provide a broader perspective on their relative strengths. To accurately identify the impact of each component on efficiency and effectiveness, we established a default setup (listed in the Appendix) and varied one component at a time, testing to observe the effects. 2.2 Metrics An ideal agent should achieve both high performance and computational efficiency. Therefore, in addition to accuracy, measured by pass@1 (solving the problem in one attempt) to evaluate effectiveness, we assess efficiency using the number of tokens taken by LLMs and associated costs. Notably, the cost of input tokens for APIs of LLMs is generally significantly lower than that of output tokens across many proprietary LLM providers. Accordingly, we compute these costs separately. The per-token pricing is sourced from official provider documentation as of May 2025. Furthermore, as different models or strategies may simultaneously impact performance and efficiency, we follow [ 27 ] and adopt the cost-of-pass metric to quantify model efficiency. The cost-of-pass metric, denoted as v(m,p) , represents the expected monetary cost of using a model m to generate a correct solution for a problem p . It is computed as the ratio of the cost of a single inference attempt, C_{m}(p) , to the success rate, R_{m}(p) : v(m,p)=\frac{C_{m}(p)}{R_{m}(p)} Here, the cost of a single inference attempt, C_{m}(p) , is defined as: C_{m}(p)=n_{\text{in}}(m,p)\cdot c_{\text{in}}(m)+n_{\text{out}}(m,p)\cdot c_{\text{out}}(m) where n_{\text{in}}(m,p) and n_{\text{out}}(m,p) are the number of input and output tokens for model m on problem p , respectively, and c_{\text{in}}(m) and c_{\text{out}}(m) are the per-token costs for input and output. The success rate R_{m}(p) is estimated by the proportion of correct responses. This metric represents the expected monetary cost of using a