Reading Recognition in the Wild
Charig Yang, Samiul Alam, Shakhrul Iman Siam, Michael J. Proulx, Lambert Mathias, Kiran Somasundaram, Luis Pesqueira, James Fort, Sheroze Sheriffdeen, Omkar Parkhi, Carl Ren, Mi Zhang, Yuning Chai, Richard Newcombe, Hyo Jin Kim, [5pt] Meta Reality Labs Research · 2025
Read Reading Recognition in the Wild by Charig Yang, Samiul Alam, Shakhrul Iman Siam, Michael J. Proulx, Lambert Mathias, Kiran Somasundaram, Luis Pesqueira, James Fort, Sheroze Sheriffdeen, Omkar Parkhi, Carl Ren, Mi Zhang, Yuning Chai, Richard Newcombe, Hyo Jin Kim, [5pt] Meta Reality Labs Research (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24848.txt.
Introduction
Yang and a team including Alam and Siam set a new task in twenty twenty-five: tell when a glasses wearer is reading, from the wearer's own view. Smart glasses need such logs to give fit help in the wild. They built a set of one hundred hours of read and non-read clips.
Three feeds feed the task: the view, eye gaze, and head pose. A transformer model takes any mix of the three feeds. Each feed helps, and blends help more, their tests show. The set also sorts read kinds, like books versus screens.
Catalog record
| CREATOR | Charig Yang, Samiul Alam, Shakhrul Iman Siam, Michael J. Proulx, Lambert Mathias, Kiran Somasundaram, Luis Pesqueira, James Fort, Sheroze Sheriffdeen, Omkar Parkhi, Carl Ren, Mi Zhang, Yuning Chai, Richard Newcombe, Hyo Jin Kim, [5pt] Meta Reality Labs Research |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 97 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-24848
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reading Recognition in the Wild Charig Yang, Samiul Alam, Shakhrul Iman Siam, Michael J. Proulx, Lambert Mathias, Kiran Somasundaram, Luis Pesqueira, James Fort, Sheroze Sheriffdeen, Omkar Parkhi, Carl Ren, Mi Zhang, Yuning Chai, Richard Newcombe, Hyo Jin Kim, [5pt] Meta Reality Labs Research Abstract To enable egocentric contextual AI in always-on smart glasses, it is crucial to be able to keep a record of the user’s interactions with the world, including during reading. In this paper, we introduce a new task of reading recognition to determine when the user is reading. We first introduce the first-of-its-kind large-scale multimodal Reading in the Wild dataset, containing 100 hours of reading and non-reading videos in diverse and realistic scenarios. We then identify three modalities (egocentric RGB, eye gaze, head pose) that can be used to solve the task, and present a flexible transformer model that performs the task using these modalities, either individually or combined. We show that these modalities are relevant and complementary to the task, and investigate how to efficiently and effectively encode each modality. Additionally, we show the usefulness of this dataset towards classifying types of reading, extending current reading understanding studies conducted in constrained settings to larger scale, diversity and realism. Figure 1 : Am I reading? The left figure shows a timeline as the user navigates the world. We aim to solve the task of reading recognition to enable AI assistants in always-on wearables. We identify three modalities: eye gaze (in colored dot patterns), RGB crop around gaze (in red box), and inertial sensors performs the task to high accuracy (with Prediction and GT shown). Images from our Reading in the Wild dataset, which features 100 hours of diverse reading and non-reading activities in real-world settings, with examples shown in the right. 1 Introduction The potential future of AI personal assistants depends on its ability to understand the physical context of the user. Smart glasses are becoming a promising device form factor capable of linking visual AI capabilities to the real world. Recently, there has been a sharp rise in the development of smart glasses, both products (Meta Ray-Ban, Amazon Echo Frames) and prototypes (Snapchat Spectacles, Halliday AI Glasses, Xreal One Pro). These all-day wearable devices enable proactive, personalized, and contextualized AI agents to perceive the world like humans do by understanding the users’ context. However, for always-on wearable glasses, due to both hardware (power, bandwidth, heat) and software (perception capability of AI agents, especially with heavy models) constraints, it is impractical to record and process every single frame over long periods of time. One solution is to have a proxy signal, so that the device can record and process key frames only when relevant. The question becomes: what forms important context of the user that the AI assistant needs to know, and how do we know when to capture them? The ability to read underpins one of, if not the most important unique modalities by which modern humans communicate, entertain each other, and learn. Reading is a key mechanism humans use to communicate with high fidelity and high information density. Reading spans a broad array of mediums, from handwritten and printed text on paper and digital displays to environmental signposts. The act of reading occurs within real-time communication with one another and today’s AI chatbots, through to reading long-form articles in books or online. Enabling AI with the ability to recognize reading is hence clearly one of the most important context signals a future AI can be enabled with to unlock truly personalized and contextually relevant AI. Given this, we ask: how can we provide future AI with the ability to know when someone is reading? This apparently simple idea underpins the ability to efficiently enable devices to know what the user has and has not read, and hence where they can assist given what it understands that the user has read. This task of reading recognition is challenging for two main reasons. First, the problem can often be ill-posed: just because a text exists in the field of view does not mean that the user is reading it (or even looking at it), which is ambiguous to solve using visual information alone. Also, the method should be efficient for real-time, always-on computation subject to the practical constraints of a wearable device. Both of these challenges render OCR-based text detection methods impractical, given the inability to solve the ambiguity and the requirements for high-resolution capture and processing. Instead, reading recognition can be used as an efficient proxy to indicate when and where it is relevant to invoke heavier models (OCR and VLMs) instead of running these models all the time. Motivated by this question, we introduce a new dataset created with Project Aria [ 12 ] glasses, which enables us to develop the contextual AI capability of detecting when a wearer is reading. We present the first-of-its-kind large-scale multimodal "Reading in the Wild" dataset, containing 100 hours of reading and non-reading videos in diverse and realistic scenarios. This dataset allows us to identify three modalities (egocentric RGB, eye gaze, head pose) that can be used to solve the task. We then present a flexible transformer model that performs the task using these modalities, either individually or combined. We show that these modalities are relevant and complementary to the task and investigate how to efficiently and effectively encode each modality, as well as the model’s ability to generalize towards unseen scenarios and perform real-time reading detection. Achieving reading recognition makes it feasible to keep a record of a user’s reading interactions with the world to build a contextually aware AI. It also enables several other applications: it allows reading assistant tools [ 41 ] in children with learning difficulties [ 5 ] and people with low vision [ 43 ] to operate in the real world; it can also be used to track whether a user has read crucial information ( e.g. signs during driving) and to measure attention and distraction while performing a task. Additionally, the dataset and method contributed in this paper can be extended to classifying different types of reading. This has been of interest in cognitive studies in reading comprehension, but they are often limited to controlled environments [ 27 , 28 , 23 , 1 , 8 ] , hence limiting its usefulness. We show that our dataset allows for reading mode and medium classification to be performed in unconstrained settings, and provide experimental results in this direction. In summary, we make the following contributions: • First, we introduce a new task of reading recognition in the wild , and demonstrate its usefulness. Unlike previous studies, we focus on in-the-wild settings and practicality towards wearable glasses. • Second, we present the first-of-its-kind large-scale egocentric multimodal Reading in the Wild dataset, which will be made publicly available, alongside a scalable protocol for data collection. • Third, we identify three modalities relevant and complementary to the task (RGB, gaze, and IMU), and develop a lightweight, flexible model that inputs these modalities either individually or in combination for reading recognition, resulting in a strong and efficient baseline for this task. • Fourth, we show that our method and dataset extend towards reading understanding, including classifying reading