Early Accessibility: Automating Alt-Text Generation for UI Icons During App Development

Sabrina Haque, Christoph Csallner · 2025

Read Early Accessibility: Automating Alt-Text Generation for UI Icons During App Development by Sabrina Haque, Christoph Csallner (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13069.txt.

Introduction

Sabrina Haque and Christoph Csallner generate icon alt-text during app building so accessibility stops being an afterthought, in work from twenty twenty-five. Screen reader users stall on icons that lack spoken labels. Fixes after release pile up debt and often never land. A developer study first asks when alt-text help is wanted. Two tuned models then draft labels from icon context.

One model reads interface metadata as text alone. The other reads the icon image plus its text context jointly. Screen structure and in-icon text feed structured prompts for accuracy. No full-screen input is needed for a good label. Comparisons with rival deep and vision-language models favour ALTICON. Quality rises while the workflow stays inside the build process.

Catalog record

CREATORSabrina Haque, Christoph Csallner
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-13069

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Early Accessibility: Automating Alt-Text Generation for UI Icons During App Development Sabrina Haque, Christoph Csallner Abstract . Alt-text is essential for mobile app accessibility, yet UI icons often lack meaningful descriptions, limiting accessibility for screen reader users. Existing approaches either require extensive labeled datasets, struggle with partial UI contexts, or operate post-development, increasing technical debt. We first conduct a formative study to determine when and how developers prefer to generate icon alt-text. We then explore the AltIcon approach for generating alt-text for UI icons during development using two fine-tuned models: a text-only large language model that processes extracted UI metadata and a multi-modal model that jointly analyzes icon images and textual context. To improve accuracy, the method extracts relevant UI information from the DOM tree, retrieves in-icon text via OCR, and applies structured prompts for alt-text generation. Our empirical evaluation with the most closely related deep-learning and vision-language models shows that AltIcon generates alt-text that is of higher quality while not requiring a full-screen input. 1. Introduction Inferring alt-text for an app icon currently either generates low-quality alt-text or requires programmers to wait until the code of a complete app screen is available. Waiting until a screen is complete causes programmers to lose important context information. Taken together, most programmers today do not use tool support for generating icon alt-text (and we have also observed this lack of tool use in our surveys). Although federal law such as Section 508 of the U.S. Rehabilitation Act and the Americans with Disabilities Act (ADA) mandates digital accessibility ( ADA.gov, 2023 ; U.S. Access Board, 2018 ) , most programmers do not add alt-text manually either. Compliance is often overlooked due to the additional manual effort required ( Di Gregorio et al., 2022 ) and time constraints and lack of prioritization during development ( Feng et al., 2021 ) . Despite established guidelines, developers often de-prioritize manual accessibility tasks due to tight deadlines, competing feature demands, and a lack of immediate impact on core functionality ( Fok et al., 2022 ; Bai et al., 2017 ) . Many apps thus ship with inadequate or missing alt-text ( Ross et al., 2018 ; Alshayban et al., 2020 ; Fok et al., 2022 ) . Meaningful alt-text however is required by screen readers such as TalkBack and VoiceOver ( Google, 2024 ; iOS, 2024 ) to make apps accessible ( Chen et al., 2021 ) . To interact with an app, almost all users with disabilities rely on screen readers (e.g., 94% of users in a recent survey ( WebAIM, 2024 ) ). Inferring the meaning of an arbitrary icon remains a fundamentally hard problem. We can only make a limited number of observations, yet have to summarize and generalize these observations to a description that captures the icon’s meaning in all possible app usage scenarios. To make this generalization, prior work has leveraged a variety of types of observation (including screenshots, view hierarchy information, and app runtime behavior). We are not aware of work that supports a programmer during development with high-quality alt-texts when the full screen information is not available yet. At a high level, recent work falls into one of the following two broad categories. In the first category are traditional deep-learning techniques that are specialized for inferring GUI element alt-texts ( Chen et al., 2020 ; Mehralian et al., 2021 ) . While they paved the way for subsequent work, these pioneering approaches are trained on relatively small data sets and struggle with generating meaningful alt-text for less-common cases. In the second category are modern vision-language models (VLMs) such as Pix2Struct ( Lee et al., 2023 ) and PaliGemma ( Beyer et al., 2024 ) . While these models have been trained on much larger datasets and fine-tuned on widget captioning tasks, they do not perform well on partial screens. These models also rely on extensive UI training data, making them resource-intensive and potentially less adaptable to unseen applications. To address these challenges, this paper proposes AltIcon , a method that integrates alt-text generation directly into the IDE, addressing accessibility gaps early during development. This paper introduces a novel application of off-the-shelf large language models to generate context-aware alt-texts for mobile app UI elements without requiring complete screen information or extensive task-specific training. Unlike existing solutions that are more capable of handling issues post-development, AltIcon is especially beneficial during the early phases of app development, where screen data—such as screenshots, UI elements, view hierarchies, and UI code—are often incomplete. To summarize, this paper makes the following major contributions. • This paper presents the first formative study on when and how developers prefer alt-text generation tools to integrate into their workflow. • This paper presents AltIcon , the first method for generating high-quality icon alt-text within the development workflow, enabling a developer to incorporate accessibility early rather than as a post-development fix. • This paper empirically compares AltIcon against the most closely related approaches and its zero-shot GPT-4o baseline on standard metrics and via a user study. • This paper analyzes AltIcon variants’ cost-performance trade-offs and the performance contribution of AltIcon ’s components and surveys potential users’ perceptions. • Our AltIcon implementation, data, fine-tuning configuration, and evaluation scripts are publicly available ( Anonymous, 2025 ) . 2. Background Android UI development has traditionally relied on XML-based layouts ( Android Studio, 2025b ) , where UI elements are defined in structured resource files and referenced from Java or Kotlin code. Accessibility attributes such as contentDescription are specified directly in XML for components like ImageButton and ImageView, which is the convention we follow for writing this paper. More recently, Android Jetpack has introduced a fully declarative UI paradigm ( Google, 2025 ) , replacing XML with composable functions written in Kotlin. Instead of defining UI elements in resource files, developers construct UIs dynamically using functions like Icon() and Image(), with accessibility metadata such as contentDescription provided within the functions. Despite these differences, the underlying accessibility principles remain unchanged, as screen readers still rely on developer-specified descriptions. AltIcon can be extended to support Jetpack with minimal modifications, adapting its alt-text generation to declarative UI structures while maintaining the same functionality. Adding alt-text is often an afterthought in development. Writing meaningful descriptions requires understanding an icon’s intended function, but developers often perceive this task as tedious and time-consuming, resulting in it being deprioritized or ignored ( Feng et al., 2021 ; Di Gregorio et al., 2022 ) . A few years ago a study of a subset of the Rico Android apps dataset ( Deka et al., 2017 ) has shown that 86% of clickable ImageViews and 55% of ImageButtons had an empty or missing contentDescription field, making them inaccessible to screen readers ( Ross et al., 2018 ) . This accessibility gap still persists today. Our analysis of the recent MUD dataset ( Feng et al., 2024 ) reveals similar shortcomings, e.g., we found 74% of Clickable ImageViews and 43% of ImageButtons still miss a contentDescription. 2.1. Context-Aware Icon Labeling Figure 1 . Example zoom out (left) vs. lower volume (right) minus buttons in Rico. Two UI icons are displayed side by side, both featuring a minus symbol. The left icon represents a ``zoom out''

Read the complete text