WebNav: An Intelligent Agent for Voice-Controlled Web Navigation

Trisanth Srinivasan, Santosh Patapati · 2025

Read WebNav: An Intelligent Agent for Voice-Controlled Web Navigation by Trisanth Srinivasan, Santosh Patapati (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13843.txt.

Introduction

Trisanth Srinivasan and Santosh Patapati build a voice-driven web agent in twenty twenty-five. Web interfaces serve users with limited mobility poorly. Screen readers and scripts lack the flexibility for dynamic pages and cannot grasp high-level goals. Their system, WebNav, aims to turn spoken commands into actions on a page.

WebNav uses two large language models in separate roles. One, the Controller, plans the next step toward the user's goal. The other, the Assistant, fills in the exact parameters to run that step. Vision from screenshots is combined with a browser extension that labels page elements. That pairing helps the agent find the right buttons and fields. Readers get a design for hands-free browsing driven by intent, not clicks.

Catalog record

CREATORTrisanth Srinivasan, Santosh Patapati
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE14 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-13843

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

WebNav: An Intelligent Agent for Voice-Controlled Web Navigation Trisanth Srinivasan, Santosh Patapati Abstract The increasing reliance on web interfaces presents many challenges for visually impaired users, showcasing the need for more advanced assistive technologies. This paper introduces WebNav, a voice-controlled web navigation agent that leverages a ReAct-inspired architecture and generative AI to provide this framework. WebNav comprises of a hierarchical structure: a Digital Navigation Module (DIGNAV) for high-level strategic planning, an Assistant Module for translating abstract commands into executable actions, and an Inference Module for low-level interaction. A key component is a dynamic labeling engine, implemented as a browser extension, that generates real-time labels for interactive elements, creating mapping between voice commands and Document Object Model (DOM) components. Preliminary evaluations show that WebNav outperforms traditional screen readers in response time and task completion accuracy for the visually impaired. Future work will focus on extensive user evaluations, benchmark development, and refining the agent’s adaptive capabilities for real-world deployment. I Introduction The increasing reliance on computers affects the visually impaired in a disproportionate manner who face lots of barriers to gaining access to and navigating the web’s increasing complexity. While screen readers provide vital functionality, they are extremely primitive and fail to address the dynamic and sophisticated nature of modern sites, leading to clumsy and frustrating user interactions [ 1 ] . This necessitates the development of more intelligent and responsive assistive technologies. WebNav meets this need with a voice-controlled web navigation agent. Based on a ReAct-inspired structure, WebNav can reason about user intent, plan action, and act upon web pages in a dynamic, context-aware fashion. This process is intended to go beyond the limitations of current assistive aids, making it easier for visually impaired individuals to have an improved experience on the web. This paper discusses WebNav’s architecture, implementation, and intentions for future evaluation. II Related Work Web accessibility has been a focus of assitive technology research for a long time. Progress has been made in developing increased accessibility for the visually impaired, but effectively navigating modern, complex web applications remains a challenge. Traditional screen readers are extremely primitive, and do not offer much benefit especially for visually impaired users seeking rapid access to information and online services. The ReAct framework [ 2 ] , which showcases the importance of synergy between reasoning and acting in language models, provides a base for building intelligent agents. ReAct’s iterative approach to task execution offers a more dynamic and adaptable method for interacting with complex environments. WebNav leverages and extends the ReAct framework for the specific application of voice-controlled web navigation. By integrating ReAct with a novel dynamic labeling system, inference layer, and assistant layer, WebNav creates a digital interface that responds intelligently to user intent, enabling more effective and efficient web exploration. While previous research includes voice interfaces for web browsing [ 1 ] and automated accessibility remediation, WebNav’s novelty lies in its intelligent agent architecture for web navigation. As Deng et al. [ 1 ] propose natural language interfaces for web accessibility, WebNav complements this by focusing on an agent-based approach for improved user experience. This is achieved through the combination of a ReAct-based controller and a dynamic labeling engine, designed for low-latency, voice-driven interaction. This agent-based approach distinguishes WebNav from existing assistive technologies, providing a more efficient and adaptable web navigation experience for the visually impaired. III System Architecture WebNav’s architecture was designed for intelligent web navigation. It consists of three main modules: the Digital Navigation Module (DIGNAV), the Assistant Module, and finally the Inference Module. These modules work together to enable the agent to perceive, reason, and act upon web content. Figure 1 illustrates this architecture. Fig. 1: WebNav System Architecture: Intelligent Agent. User voice commands initiate a process within WebNav. The DIGNAV Controller uses the ReAct framework for high-level planning, the Assistant Module refines commands into actionable steps, and the Inference Module executes these actions, completing a cycle of perception, planning, and action for voice-controlled web navigation. III-A Digital Navigation Module DIGNAV serves as the cognition for WebNav, designed for intelligent web exploration and interaction. Its components are structured to enable effective decision-making: The voice input and processing subsystem receives user commands via microphone and converts them into text using the Whisper speech recognition model [ 3 ] . This text serves as the initial input for the agent. The DIGNAV Controller is the reasoning component, implementing a modified ReAct framework and utilizing a Google Generative AI model (‘gemini-2.0-flash-thinking-exp-01-21‘). The controller operates in an iterative ”Reason” and ”Act” cycle. In the ”Reason” phase, it analyzes the user’s goal, the current web page state (from labeled and unlabeled screenshots), and the history of previous interactions. Based on this analysis, the controller transitions to the ”Act” phase, outputting a high-level command like ”Click [5]” or ”Type [3] search query.” The ReAct framework allows the controller to adapt its reasoning and actions dynamically based on the evolving web page context and the outcomes of previous actions. The Assistant module refines the strategic commands from the DIGNAV Controller, transforming them into concrete, executable actions. This module, also powered by a Gemini model, acts as a tactical planner. It takes the high-level intent from the Controller and generates a JSON payload specifying the function to be executed (e.g., ”Click”, ”Type”) and the necessary parameters (e.g., element number, text). III-A 1 Dynamic Labeling Engine The dynamic labeling engine is a component that provides WebNav with a representation of the web page, enabling interaction. Implemented as a browser extension, it dynamically assigns unique labels to interactive elements on the page. When activated, the engine scans the web page to identify interactive elements. It checks their visibility and size to ensure only relevant elements are considered. Each identified element receives a visual overlay with a numerical label, making it easy to reference. These overlays adjust dynamically as the page changes, thanks to a monitoring system. A mapping system links each number to its corresponding element, allowing user interactions to be translated into actions. The extension’s core functionality is managed through its underlying Chrome architecture, ensuring seamless deployment and activation. ⬇ function createLabel ( element , rect , color ) { const overlay = document . createElement ( "div" ); overlay . className = "element-overlay" ; overlay . style . position = "absolute" ; overlay . style . left = ‘ $ { rect . left + window . scrollX } px ‘; overlay . style . top = ‘ $ { rect . top + window . scrollY } px ‘; overlay . style . width = ‘ $ { rect . width } px ‘; overlay . style . height = ‘ $ { rect . height } px ‘; overlay . style . borderColor = color ; overlay . style . backgroundColor = ‘ $ { color }22‘; const label = document . createElement ( "div" ); label . className = "element-label" ; const currentNumber = counter ++; label . textContent = currentNumber . toString (). padStart (2, "0" ); label . style . backgroundColor = color ; return overlay ; } Listing 1: Label Creation Excerpt (JavaScript) The Inference module executes the actions planned by the DIGNAV Controller and refined by the Assistant. Using

Read the complete text