WebSight : A Vision-First Architecture for Robust Web Agents
Tanvir Bhathal*, Asanshay Gupta* · 2025
Read WebSight : A Vision-First Architecture for Robust Web Agents by Tanvir Bhathal*, Asanshay Gupta* (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-16987.txt.
Introduction
Tanvir Bhathal and Asanshay Gupta present in 2025 WebSight, a web agent that browses by looking at the screen rather than reading the page's code. Most agents lean on HTML structure, which breaks when sites are messy or scripts hide the real layout; humans cope by sight alone.
At the center sits a vision-language model, fine-tuned on web interface data, that finds buttons and fields in screenshots. Around it, separate agents plan, reason, act, and verify, sharing an episodic memory. On a clicking benchmark the sight model led its class. The design aims at agents that work on the web as people see it.
Catalog record
| CREATOR | Tanvir Bhathal*, Asanshay Gupta* |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 112 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-16987
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
WebSight : A Vision-First Architecture for Robust Web Agents Tanvir Bhathal*, Asanshay Gupta* Abstract We introduce WebSight , a vision-based autonomous web agent, designed to interact with web environments purely through visual perception, eliminating dependence on HTML or DOM-based inputs. Central to our approach we introduce our new model, WebSight -7B, a fine-tuned vision-language model optimized for UI element interaction, trained using LoRA on a web-focused subset of the Wave-UI-25K dataset. WebSight integrates this model into a modular multi-agent architecture, comprising planning, reasoning, vision-action, and verification agents, coordinated through an episodic memory mechanism. WebSight -7B achieves a top-1 accuracy of 58.84% on the Showdown Clicks benchmark, outperforming several larger generalist models while maintaining lower latency. The full WebSight agent achieves a 68.0% success rate on the WebVoyager benchmark, surpassing systems from labs such as OpenAI (61.0%) and HCompany (Runner H, 67.0%). Among tasks completed, WebSight answers correctly 97.14% of the time, indicating high precision. Together, WebSight and WebSight -7B establish a new standard for interpretable, robust, and efficient visual web navigation. 1 Introduction Autonomous web agents capable of performing complex web navigation tasks—such as automated form filling, online shopping, and dynamic information retrieval—have emerged as a critical area of study within artificial intelligence. Over recent years, substantial progress has been driven by leveraging large language models (LLMs), enabling browser agents to interpret web content primarily through textual representations like HTML code, Document Object Model (DOM) trees, and accessibility metadata nakano2021webgpt ; xu2023wizardlm ; yang2023sphynx . Despite their successes, these approaches present critical challenges when confronted with real-world scenarios. Specifically, websites frequently feature incomplete or incorrect metadata, dynamic layouts, and complex designs that degrade the reliability of structurally-dependent agents sun2022recovering ; xu2022focal . Furthermore, the computational demands of processing extensive textual and structured inputs limit their scalability and interpretability, hindering practical deployment zhang2023autoagents ; li2024agentbench ; ganguli2022predictability . Figure 1 : The WebSight Architecture In contrast, human users rely almost exclusively on visual perception, effortlessly recognizing actionable interface elements like buttons, input fields, and navigation bars based solely on visual layout and design cues, irrespective of underlying metadata donker2002visual ; zhang2021screen2words . Inspired by this natural modality, we present two tightly integrated contributions: WebSight , a vision-based autonomous browser agent; and WebSight -7B, a task-specific vision-language model trained to identify and interact with user interface elements directly from rendered web screenshots. The WebSight agent is built on a robust multi-agent orchestration framework that mimics human cognitive processes. At its core are dedicated planning agents, which devise comprehensive strategies outlining task sequences and continually track progression towards goals kaelbling1998planning ; russell2010artificial . Concurrently, specialized reasoning agents perform detailed analyses to pinpoint and articulate precise subsequent actions necessary for task advancement bommasani2021opportunities ; ganguli2022predictability . The critical component of our architecture, the vision agent, leverages the WebSight -7B model qin2025ui ; hu2022lora to make decisions on actions in the browser. Post-action, dedicated verification agents rigorously evaluate the resultant webpage state changes to ascertain accuracy and effectiveness of each interaction sun2022recovering ; xu2022focal . This integrative process is further bolstered by an episodic memory mechanism, dynamically updating and iteratively refining agent strategies through continuous verification cycles until completion criteria are achieved shi2022learning ; li2024agentbench ; zhang2023autoagents . Collectively, this orchestrated multi-agent architecture closely mirrors human cognitive and perceptual workflows, significantly enhancing interpretability, adaptability, and robustness donker2002visual ; zhang2021screen2words . At the core of this system, WebSight -7B is a fine-tuned version of UI-TARS that we adapt to the web domain qin2025ui . Trained on augmented web-based GUI datasets using LoRA fine-tuning, WebSight -7B demonstrates substantial improvements in English-language understanding and precision UI interaction over generalist vision-language models hu2022lora . We show that the combination of architectural modularity and a domain-specialized foundation model enables both strong performance and practical deployability. To validate our contributions, we evaluate WebSight and WebSight -7B on two challenging benchmarks: WebVoyager and Showdown Clicks he2024webvoyager ; skyvern2024webvoyager ; showdown2025 . WebSight achieves a 68.0% success rate on WebVoyager, outperforming comparable agents by 1–7 percentage points. On the Showdown Clicks benchmark, WebSight -7B attains a top-1 accuracy of 58.84%, surpassing a range of larger general-purpose vision-language models by 4–7 points. These results underscore the benefits of a vision-first architecture and domain-specialized model design for effective and efficient web interaction. Through WebSight , we introduce a dual innovation: a vision-first web agent and an accompanying domain-optimized vision-language model. Together, advance autonomous web agent research by explicitly integrating human-like visual perception into agent design. Our approach offers a path toward robust, interpretable, and computationally efficient autonomous agents capable of generalizing across diverse web environments, aligning closely with human browsing behavior and intuitions. We publicly share WebSight ’s code on Github : https://github.com/SuperAce100/websight and WebSight -7B on HuggingFace: https://huggingface.co/tanvirb/websight-7B . 2 Related Works Early web navigation agents predominantly relied on reinforcement learning techniques, demonstrating basic task execution capabilities lauer2004learning ; tesauro2005online ; branavan2009reinforcement . The advent of deep reinforcement learning further enhanced agent capabilities, enabling more sophisticated interaction strategies mnih2015human ; schulman2017proximal ; hessel2018rainbow ; vinyals2019grandmaster . However, these agents often required extensive training environments and struggled with generalization to unseen websites. More recent advancements in large language models (LLMs) such as GPT series brown2020language ; ouyang2022training ; openai2023gpt4 have significantly improved autonomous agents’ abilities to generalize and handle complex instructions zhou2023webarena ; liu2023webagent ; xu2023wizardlm . Nonetheless, these approaches remain heavily reliant on structured textual inputs such as DOM trees and accessibility metadata, limiting their robustness against incomplete or inaccurate metadata sun2022recovering ; xu2022focal ; zhang2023autoagents ; li2024agentbench . Parallel to textual methods, computer vision approaches have significantly advanced, particularly in semantic segmentation and object detection he2017mask ; redmon2018yolov3 ; chen2017deeplab ; carion2020endtoend ; zhang2020resnest . Recent progress in visual transformer architectures and self-supervised learning has further pushed the boundary of vision-based perception capabilities dosovitskiy2020image ; caron2021emerging ; radford2021learning ; bao2022beit ; wang2022image . Integrating visual perception into web navigation has also garnered attention. Early work utilized vision heuristics for webpage understanding cai2001extracting ; kumar2007extracting ; dong2010towards . More recent methods have employed deep learning models to directly interpret visual webpage layouts, significantly improving navigation robustness zhang2021screen2words ; kirillov2023segment ; ravi2024sam2 ; liu2023webui . Our multi-agent orchestration approach builds upon foundational theories in multi-agent systems and planning kaelbling1998planning ; russell2010artificial ; wooldridge2009introduction , cognitive architectures anderson2009can ; laird2012soar , episodic memory integration blundell2016model ; pritzel2017neural , and human-inspired verification strategies mnih2015human ; silver2016mastering ; lecun2022path to achieve more robust, interpretable, and efficient autonomous web agents. 3 Data UI Dataset Synthetic Augmentation Original Data • Image + bbox • Type: button • Text: ”Reminder” • Purpose: reminders Templates ”Click {element} to {purpose}” ”Tap {type} for {function}” ”Select {name}” Augmented Prompts ”Click button to set reminders” ”Tap reminder for management” ”Select reminder button” Figure 2 : Synthetic augmentation pipeline for UI vision-language model training. Original structured UI data is processed through natural language templates to generate diverse instruction variations. We utilize AgentSea’s Wave-UI-25K dataset waveui25k . Wave-UI-25K comprises of 24,978 entries. Of this, we use the subset of 22,994 web-based screenshots with annotated browser interaction elements, as we are not focusing on alternative subsets like mobile and desktop. Each dataset entry includes a screenshot, the bounding box of a particular UI element within it, OCR extracted