ROSA: Addressing text understanding challenges in photographs via ROtated SAmpling
Hernán Maina, Guido Ivetta, Mateo Lione Stuto, Julian Martin Eisenschlos, Jorge Sánchez, Luciana Benotti · 2025
Read ROSA: Addressing text understanding challenges in photographs via ROtated SAmpling by Hernán Maina, Guido Ivetta, Mateo Lione Stuto, Julian Martin Eisenschlos, Jorge Sánchez, Luciana Benotti (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-03665.txt.
Introduction
Hernán Maina and five co-authors help vision systems read sideways text in photos, in a research paper from 2025. People with poor vision could use question-answering tools to read text around them. But their photos often frame text at odd angles, and current models struggle with that.
Interviews with visually impaired users confirmed that tilted text is a common problem. Standard test sets, built from photos by sighted users, rarely include it. Their method, ROSA, tries the image at several rotations when answering. On the best model tested, it beat the standard approach by a clear margin on rotated text.
Catalog record
| CREATOR | Hernán Maina, Guido Ivetta, Mateo Lione Stuto, Julian Martin Eisenschlos, Jorge Sánchez, Luciana Benotti |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-03665
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ROSA: Addressing text understanding challenges in photographs via ROtated SAmpling Hernán Maina, Guido Ivetta, Mateo Lione Stuto, Julian Martin Eisenschlos, Jorge Sánchez, Luciana Benotti Abstract Visually impaired people could benefit from Visual Question Answering (VQA) systems to interpret text in their surroundings. However, current models often struggle with recognizing text in the photos taken by this population. Through in-depth interviews with visually impaired individuals, we identified common framing conventions that frequently result in misaligned text. Existing VQA benchmarks primarily feature well-oriented text captured by sighted users, under-representing these challenges. To address this gap, we introduce ROtated SAmpling ( R O S A ROSA ), a decoding strategy that enhances VQA performance in text-rich images with incorrectly oriented text. R O S A ROSA outperforms G r e e d y Greedy decoding by 11.7 absolute points in the best-performing model. 1 Introduction Extracting text from the surroundings is essential for people with visual impairments. Large multimodal models (LMMs), which possess visual and linguistic processing capabilities, have shown impressive results in tasks such as VQA. They offer great potential but still face real-world challenges that limit their effectiveness in assistive applications for this population. Although advances in deep learning have improved the detection and recognition of text in arbitrary orientations and shapes He et al. 2016 ; He et al. 2018 ; Yao et al. 2016 ; Bazazian et al. 2017 ; Ma et al. 2018 ; Liao et al. 2018 ; Zhang et al. 2024 , such work typically addresses challenges posed by pictures taken by sighted people in natural scenes—such as the diversity of fonts, sizes, colors, and lighting. Bigham et al. 2010 highlight that nearly a quarter of the questions posed by individuals with visual impairments require reading text present in images. Yet, the specific challenges of reading text in pictures taken by people with visual impairments remain an under-researched area. Figure 1: Examples of image pairs and associated visual questions retrieved from the VizWiz-VQA Gurari et al. 2018 dataset, highlighting challenges such as recognizing text in more than one orientation, addressing occlusions caused by varying surface textures, and handling unconventional framing. These challenges are illustrated from left to right in the three images, respectively. Figure 1 illustrates the challenges present in pictures taken by people with visual impairments. The images were extracted from the VizWiz-VQA Gurari et al. 2018 dataset. The photo on the left illustrate one such challenge: it contains text in different orientations—the yellow text that says spice is upside down, while the response is written vertically. As a result, no single rotation can make all the text in the picture correctly oriented. The image in the middle also contains text which is not correctly oriented with respect to the hand that holds it. As we shall see, people with visual impairments often struggle to correctly orient symmetric objects. This image also presents distorted text and shadows caused by the material of the bag and the way it is being held. Finally, the image on the right contains only a partial answer—the word pasta is only partially visible due to incorrect framing. Sighted people tend to frame pictures by centering the object of interest and leaving some margin around it, but this is not easy to achieve without sight, as our interview process confirms. In this paper we pose three main questions: (1) How do incorrectly oriented text affect the performance of models in VQA tasks? , (2) What strategies can mitigate these limitations? , and (3) What factors influence the framing of pictures, and their contained texts, when taken by individuals with visual impairments? . Our work undertakes the two first questions from complementary perspectives. First, we show that incorrect text orientations in the text that contains the answers significantly impact all models performance, with an average performance drop of approximately 7.7 absolute points when using the conventional Greedy decoding. Second, we introduce ROtated SAmpling ( ROSA ), a simple yet effective strategy that combines sampling-based decoding with image rotations, outperforming conventional Greedy decoding by 11.7 absolute points on the best-performing model. To address the third question we perform in depth interview with people with visual impairments. We evaluate our proposal using subsets of the VizWiz-VQA dataset, which consists of pictures taken by visually impaired people during their daily life. Those pictures, together with a question (as illustrated in Figure 1 ), were sent through an app to a person that could answer the questions. The subsets focus on images containing incorrectly oriented text. The orientations are based on interviews with blind and low-vision individuals, providing insights into how intuitive strategies and conventions learned by them influence the way they frame and capture pictures. It is important to emphasize that the visual perception of individuals with visual impairments and the rules governing it are shaped through interactions with their environment and experiences provided by family members, educators, and other mediators Lewis 1969 ; Ludíková and Finková 2012 ; Majerova 2017 . Our main contributions are as follows: • We quantify the weaknesses of state-of-the-art multimodal models in VQA tasks that require answering questions about photographs where reading incorrectly oriented text is necessary. • We identified intuitive patterns and conventions learned by people with visual impairments regarding the orientation of objects and their textual elements, and how these influence the way they frame and capture photographs. Based on these findings, we created evaluation subsets derived from the VizWiz-VQA dataset, incorporating systematic variations through rotated images. • We design and evaluate ROtated SAmpling ( ROSA ), a decoding strategy that combines sampling-based generation with image rotations. Our results demonstrate its effectiveness in consistently improving the performance of several state-of-the-art multimodal models on images containing incorrectly oriented text. The remainder of the article is structured as follows: Section 2 reviews prior Optical Character Recognition (OCR)-based VQA models, and techniques relevant to our proposal. Section 3 presents key findings from interviews with visually impaired individuals on their challenges with text in images. Section 4 details the evaluation datasets, informed by the interviews, and the metrics used. Section 5 introduces our ROSA strategy in detail, its operation, and design motivation. Section 6 presents and analyzes the experimental results, including the discussion of the interview insights and the ROSA ’s performance comparison to other strategies. Finally, Section 7 concludes the paper, highlighting the importance of visually impaired individuals’ needs in VQA design. 2 Related work VQA models designed to read and understand text in images have received increasing attention in recent years. Early approaches, such as LoRRA Singh et al. 2019 , extended standard VQA architectures by incorporating Optical Character Recognition (OCR) tokens into the answer generation process. Subsequent works, like M4C Hu et al. 2020 , introduced multimodal fusion mechanisms to better integrate text and image features, improving performance on datasets such as TextVQA Singh et al. 2019 , ST-VQA Biten et al. 2019 and OCR-VQA Mishra et al. 2019 . More recently, models such as TAP Yang et al. 2021 and Pix2Struct Lee et al. 2023 have leveraged transformer-based architectures and pretraining on large-scale datasets to enhance OCR-based VQA capabilities. However, these models are typically trained and