Aegis: Taxonomy and Optimizations for Overcoming Agent-Environment Failures in LLM Agents

Kevin Song, Anand Jayarajan, Yaoyao Ding, Qidong Su, Zhanda Zhu, Sihang Liu, Gennady Pekhimenko · 2025

Read Aegis: Taxonomy and Optimizations for Overcoming Agent-Environment Failures in LLM Agents by Kevin Song, Anand Jayarajan, Yaoyao Ding, Qidong Su, Zhanda Zhu, Sihang Liu, Gennady Pekhimenko (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-19504.txt.

Introduction

Kevin Song and co-authors chart in 2025 how chat agent work breaks at the seam with its world, not just in the agent's own calls. Past work buffs the agent and skips the stage it acts on. Their set draws one hundred forty two runs cross five test worlds.

Six break kinds sort the fails from over three thou sand turns of play. Three stage fixes then rise from the chart: richer stage sight, shared math off load, and bold pre runs of next acts. Win rates lift by near seven to twelve per cent with no change to the agent or its core model.

Catalog record

CREATORKevin Song, Anand Jayarajan, Yaoyao Ding, Qidong Su, Zhanda Zhu, Sihang Liu, Gennady Pekhimenko
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-19504

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Aegis: Taxonomy and Optimizations for Overcoming Agent-Environment Failures in LLM Agents Kevin Song, Anand Jayarajan, Yaoyao Ding, Qidong Su, Zhanda Zhu, Sihang Liu, Gennady Pekhimenko Abstract . Large Language Models (LLMs) agents augmented with domain tools promise to autonomously execute complex tasks requiring human-level intelligence, such as customer service and digital assistance. However, their practical deployment is often limited by their low success rates under complex real-world environments. To tackle this, prior research has primarily focused on improving the agents themselves, such as developing strong agentic LLMs, while overlooking the role of the system environment in which the agent operates. In this paper, we study a complementary direction: improving agent success rates by optimizing the system environment in which the agent operates. We collect 142 agent traces (3,656 turns of agent-environment interactions) across 5 state-of-the-art agentic benchmarks. By analyzing these agent failures, we propose a taxonomy for agent-environment interaction failures that includes 6 failure modes. Guided by these findings, we design Aegis, a set of targeted environment optimizations: 1) environment observability enhancement, 2) common computation offloading, and 3) speculative agentic actions. These techniques improve agent success rates on average by 6.7 − 12.5 % 6.7-12.5\% , without any modifications to the agent and underlying LLM. 1. Introduction Large Language Models (LLMs) are increasingly being deployed as autonomous agents to perform complex, multi-turn real-world tasks. LLM agents are equipped with tools, usually in the form of functions, to interact with the outside world, such as database queries ( spider1 ; spider2 ; text2sql_survey ) , file system operations ( bfcl ; sweagent ; agent_fs ) , and web searches ( agentoccam ; webarena ; chatgpt_agent ) . The combination of powerful foundation models and domain-specific tools enables autonomous systems that can perform complex real-world tasks, such as customer service ( taubench ; proxyllm ) , business data management ( crmarena ; crmarena_pro ) , and digital assistant ( llm_assist ; personal_llm ) . Despite this promise, practical agent deployments suffer from low task success rates ( crmarena_pro ; llms_lost ; llm_distraction ; agent_misalign ; agent_safe ; andriod_arena ) . Agent failures occur when agents perform tasks incorrectly, producing misleading output or undesirable side effects, requiring costly human interventions ( agent_fail_cost1 ; agent_fail_cost2 ; agent_fail_cost3 ) . To address this challenge, prior works focus on improving the agent/LLM, such as building foundation LLMs with stronger agentic abilities ( o3 ; gemini25 ) or fine-tuning LLMs for specific workloads ( xlam ; xlam3 ; xlam2 ) . We refer to this paradigm as agent-for-system . However, while state-of-the-art LLMs demonstrate strong intrinsic abilities in mathematics and logic ( o1 ; o3 ; dsr1 ; gemini_deep_think ) , their performance on real-world agentic tasks remains limited. OpenAI o3 ( o3 ) , a model optimized for agentic use cases, improves success rates on customer service workloads ( taubench ) by only 2-3% ( o3 ) over its predecessor o1, illustrating that strengthening models alone has limited effectiveness. To understand why agents fail despite strong underlying LLMs, we conducted a preliminary analysis of failure traces from state-of-the-art agentic benchmarks ( bfcl ; taubench ) . We found that failures stem not only from flawed LLM reasoning, but also from the agent’s interaction with its surrounding environment, such as inability to discover information, follow workload constraints, and process large volumes of data ( spider2 ; crmarena_pro ; why_multiagent_fail ) . This led us to ask: How significant is the system environment for agent reliability, and how much improvement can environment optimizations achieve? In this paper, we investigate this complementary but largely overlooked direction: optimizing the system environment to improve agent reliability, an approach which we term system-for-agent. Table 1. Taxonomy of agent system-interaction failures and corresponding environment optimizations. Category Subcategory Description Optimization Exploration Failures ( § 4.1 ) State-space Navigation Failure Agent fails to navigate the environment to retrieve all necessary data required to complete the task. Environment lookahead ( § 5.1 ) State Awareness Failure Agent has incorrect understanding about its current position within the environment. Explicit agent state changes ( § 5.1 ) Exploitation Failures ( § 4.2 ) Tool Output Processing Failure Agent makes computational errors (e.g., comparisons, ranking) when processing information gathered from tool outputs. Offload common computations ( § 5.2 ) Domain Rule Violation Agent fails to follow domain rule by either performs forbidden actions or incorrectly blocks valid actions. Offload domain rule validations ( § 5.2 ) User Instruction Following Failure Agent fails to follow user’s specific instructions as requested. No direct environment optimizations Resource Exhaustion ( § 4.3 ) – Agent fails due to exceeding allocated maximum number of turns or tokens before task completion. Speculative agentic actions ( § 5.3 ) However, designing effective environments for agents is non-trivial, as real-world system environments are complex and vary widely across different domains. To bridge this gap, we present an in-depth study of agent-environment interactions and a systematic methodology for optimizing system environments to improve agent reliability. We collect agent failure traces comprising 142 failed agent tasks and 3,656 turns of agent interactions. To precisely localize agent failures, we introduce a subtask-based abstraction inspired by Hierarchical Task Networks (HTNs) from the automated planning literature ( htn ; htn2 ) . Using this abstraction, we annotate the collected failure traces to identify the specific subtask where each failure first occurred. Based on these annotations, we propose a taxonomy of agent-environment interaction failures, summarized in Table 1 . We categorize failures into three main types: 1) Exploration failures , which occur when agent fails to gather all information required to complete the task, 2) Exploitation failures , which arise when agent incorrectly processes information it has collected, and 3) Resource exhaustion , which accounts for failures due to exceeding the pre-allocated turns/token limit. Based on our analysis, we propose Aegis, a set of system-level optimizations to address the identified failure modes, as summarized in Table 1 . To mitigate exploration failures, we enhance environment observability through expanding the agent’s observation window and explicitly communicating state changes. To reduce exploitation failures, we offload deterministic reasoning operations such as ranking and rule-checking from the agent to be performed in the environment instead. Lastly, we introduce speculative agentic actions to address resource exhaustion by preemptively bundling related tool calls, reducing turn count and token consumption. We evaluate the effectiveness of our environment optimizations across 5 state-of-the-art agent benchmarks, achieving 6.7-12.5\% average success rate improvements on average without any modifications to the agent. For context, this improvement is comparable to or greater than those typically seen between major LLM model generations, which are typically a few percent ( o3 ; claude4 ; gpt41 ) . We further validate these improvements using a model fine-tuned for agentic workloads: xLAM-2 8B ( xlam3 ) . Our environment optimizations increase success rates by 3.3-8.8\% on average, demonstrating that system-for-agent complements model fine-tuning. Lastly, we observe an unexpected benefit of environment optimizations: the monetary cost of LLM inference API is reduced by 7.1-17.7\% as a result of more efficient agent-environment interactions. In summary, we make the following contributions: \bullet We collect and analyze 142 failed tasks and 3,656 agent interactions across 5 diverse agentic workloads. Based on this analysis, we

Read the complete text