Learning to Contextualize Web Pages for Enhanced Decision Making by LLM Agents
Dongjun Lee, Juyong Lee, Kyuyoung Kim, Jihoon Tack, Jinwoo Shin, Yee Whye Teh, Kimin Lee, KAIST AI, University of Oxford · 2025
Read Learning to Contextualize Web Pages for Enhanced Decision Making by LLM Agents by Dongjun Lee, Juyong Lee, Kyuyoung Kim, Jihoon Tack, Jinwoo Shin, Yee Whye Teh, Kimin Lee, KAIST AI, University of Oxford (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10689.txt.
Introduction
Dongjun Lee and co-authors teach language models to rewrite messy web pages into a form that software agents can act on, in a 2025 paper. Agents that browse for users often fail at plain tasks because real pages are cluttered and hard to parse. Their framework, LCoW, splits the job: one module reads the page and restates it clearly, and a second module decides and acts on that clean version.
Paired with several well-known agents, the rewriter lifted success rates on a work-task benchmark by about a sixth for closed models and a quarter for open ones. On a shopping benchmark, the boosted agent beat human experts, a first in the paper's account. The gains held across model sizes. The authors argue that reading well, apart from deciding well, is a skill worth training on its own, and they share their materials in public.
Catalog record
| CREATOR | Dongjun Lee, Juyong Lee, Kyuyoung Kim, Jihoon Tack, Jinwoo Shin, Yee Whye Teh, Kimin Lee, KAIST AI, University of Oxford |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-10689
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning to Contextualize Web Pages for Enhanced Decision Making by LLM Agents Dongjun Lee, Juyong Lee, Kyuyoung Kim, Jihoon Tack, Jinwoo Shin, Yee Whye Teh, Kimin Lee, KAIST AI, University of Oxford Abstract Recent advances in large language models (LLMs) have led to a growing interest in developing LLM-based agents for automating web tasks. However, these agents often struggle with even simple tasks on real-world websites due to their limited capability to understand and process complex web page structures. In this work, we introduce LCoW, a framework for L earning language models to Co ntextualize complex W eb pages into a more comprehensible form, thereby enhancing decision making by LLM agents. LCoW decouples web page understanding from decision making by training a separate contextualization module to transform complex web pages into comprehensible format, which are then utilized by the decision-making agent. We demonstrate that our contextualization module effectively integrates with LLM agents of various scales to significantly enhance their decision-making capabilities in web automation tasks. Notably, LCoW improves the success rates of closed-source LLMs (e.g., Gemini-1.5-flash, GPT-4o, Claude-3.5-Sonnet) by an average of 15.6%, and demonstrates a 23.7% average improvement in success rates for open-source LMs (e.g., Llama-3.1-8B, Llama-3.1-70B) on the WorkArena benchmark. Moreover, the Gemini-1.5-flash agent with LCoW achieves state-of-the-art results on the WebShop benchmark, outperforming human experts. The relevant code materials are available at our project page: https://lcowiclr2025.github.io . † † footnotetext: ∗ Equal contribution 1 Introduction Figure 1: Success rate of the Gemini-1.5-flash agent on 40 WorkArena tasks. We selected a subset of 40 tasks by choosing the first 40 tasks based on the task indices. When the agent leverages observations contextualized by GPT-4o (yellow), its success rate improves by 31%, with further improvements achieved with our method (green). Large language models (LLMs) have demonstrated strong potential in automating web tasks by treating web browsing as a sequential decision-making process, where web pages serve as observations and user interactions, such as clicking and typing, function as actions ( Yao et al., 2022a ; Yao et al., 2022b ) . Various approaches have been developed to enhance the performance of LLM agents in these tasks. One such method involves fine-tuning open-source LLMs using demonstration data from web browsing tasks ( Furuta et al., 2023 ; Lai et al., 2024 ) . While promising, this approach requires extensive data collection and significant computational resources for effective fine-tuning. Alternatively, several studies have utilized advanced closed-source LLMs, such as GPT-4o ( OpenAI, 2024 ) , with carefully designed prompting techniques ( Drouin et al., 2024 ; Zhou et al., 2023a ; Sodhi et al., 2024 ; Pan et al., 2024 ) . By leveraging the general world knowledge and reasoning capabilities of these models, the methods enhance web automation but at the cost of reduced controllability. However, despite the advancements, state-of-the-art LLM agents often struggle to process complex raw web content, such as HTML and accessibility trees, posing significant challenges for their effective use in web task automation. While LLMs excel in tasks that require detailed reasoning, such as solving mathematical problems or coding, we hypothesize that their underperformance in seemingly simple decision-making tasks like web browsing is not due to a lack of decision-making capabilities but rather to difficulties in understanding and processing complex web page observations. To validate our hypothesis, we conducted an initial experiment that demonstrated an LLM agent based on Gemini-1.5-flash can achieve substantial improvements in web browsing tasks when equipped with a module designed to contextualize complex web page observations (i.e., contextualization module). This module enhances task performance by removing irrelevant UI elements and highlighting key components with explanations, thereby simplifying the decision-making process. We evaluated the performance of this agent on 40 tasks from WorkArena ( Drouin et al., 2024 ) , a benchmark designed to assess web agents on real-world, enterprise-related websites. As shown in Figure 1 , utilizing GPT-4o as the contextualization module (yellow) resulted in a 31% absolute improvement in the agent’s success rate compared to direct processing of raw observations (red). These results support our hypothesis that the difficulty in understanding web pages is a major bottleneck for LLM-based web agents. Figure 2: (Top) In the conventional pipeline, LLM agents decide on the next action based on raw, complex web page observations (e.g., HTML, accessibility trees), which often hinder accurate decision making. (Bottom) In our proposed pipeline, a contextualization module transforms these complex web page observations into a more comprehensible format, thereby enabling LLM agents to make more accurate decisions by enhancing their understanding of the web page. In this work, we propose LCoW, a framework that includes a contextualization module and a training algorithm to fine-tune this module to enhance the decision-making capabilities of LLM agents in web automation. As illustrated in Figure 2 , the contextualization module transforms complex web page observations into a comprehensible format, enabling LLM agents to make more accurate decisions. Furthermore, to enable the contextualization module to provide context more grounded in real websites, we propose an iterative algorithm designed to train the contextualization module. Our training algorithm consists of three phases: (i) trajectory collection, (ii) sampling contextualized observations, and (iii) updating the contextualization module. For each observation from the collected trajectories, we generate multiple contextualized observations using the current contextualization module. Each observation is then assigned a reward based on whether a set of LLM agents can accurately predict the correct action given the contextualized observation. Finally, we select the one with the highest reward as the target and train the contextualization module to maximize the likelihood of the target given the original raw observation. As demonstrated in our initial experiment using the Gemini-1.5-flash (see Figure 1 ), LCoW significantly enhances the decision-making capabilities of LLM agents, even beyond the improvements seen with state-of-the-art LLMs like GPT-4o used as a contextualization module. In our experiments, we conduct comprehensive evaluations of our proposed approach on three well-established benchmarks for evaluating agent performance in web environments: WebShop ( Yao et al., 2022a ) , WorkArena ( Drouin et al., 2024 ) , and WebArena ( Zhou et al., 2023a ) . In detail, we first demonstrate that LCoW significantly enhances the overall performance of LLM agents with varying scales in the main experiment. Furthermore, we analyze the behavior of the contextualization module in detail, including qualitative examination of the observation refined by the contextualization module, comparison with training LLM agents using behavior cloning, and generalization ability. 2 Background In this section, we describe the formulation of web browsing as a sequential decision-making problem and the use of LLMs as decision-making agents. Web browsing can be formulated as a Partially Observable Markov Decision Process (POMDP), defined by \langle\mathcal{S},\mathcal{O},\mathcal{A},T,\mathcal{R}\rangle . The state s_{t}\in\mathcal{S} represents the internal configuration of the web browser at time step t , which is only partially observable. The observation o_{t}\in\mathcal{O} corresponds to the web page rendered by the web browser given s_{t} , which can take various forms (e.g., screenshot, HTML, accessibility trees). The action space \mathcal{A} is the set of all possible interactions with the UI elements (e.g., clicking, typing). The state transition function T defines the probability of transitioning from state s to state s^{\prime} after performing an action a_{t}\in\mathcal{A} , such as