Beyond Screenshots: Evaluating VLMs’ Understanding of UI Animations
Chen Liang, Xirui Jiang, Naihao Deng, Eytan Adar, Anhong Guo · 2026
Read Beyond Screenshots: Evaluating VLMs’ Understanding of UI Animations by Chen Liang, Xirui Jiang, Naihao Deng, Eytan Adar, Anhong Guo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-26148.txt.
Introduction
Liang and four colleagues test whether vision-language models understand user-interface animations, a channel static screenshots miss. They build AniMINT, a dataset of three hundred annotated animation videos. They test leading models on motion effects, animation purposes, and animation meaning. Models detect primitive motion well but grasp higher-level meaning inconsistently, lagging humans. Motion, context, and perceptual cues point to the key bottlenecks.
Catalog record
| CREATOR | Chen Liang, Xirui Jiang, Naihao Deng, Eytan Adar, Anhong Guo |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62444 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.26148
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Screenshots: Evaluating VLMs’ Understanding of UI Animations Chen Liang, Xirui Jiang, Naihao Deng, Eytan Adar, Anhong Guo Abstract AI agents operating on user interfaces must understand how interfaces communicate state and feedback to act reliably. As a core communicative modality, animations are increasingly used in modern interfaces, serving critical functional purposes beyond mere aesthetics. Thus, understanding UI animation is essential for comprehensive interface interpretation. However, recent studies of Vision Language Models (VLMs) for UI understanding have focused primarily on static screenshots, leaving it unclear how well these models handle dynamic UI animations. To address this gap, we created AniMINT , a novel dataset of 300 densely annotated UI animation videos. We systematically evaluate state-of-the-art VLMs on UI animation understanding, including their abilities to perceive the animation effects, identify animation purposes, and interpret animation meaning. Our results show that VLMs can reliably detect primitive motion. However, their high-level animation interpretation remains inconsistent, with substantial gaps relative to human performance. Finally, we use Motion, Context, and Perceptual Cues (MCPC) to probe factors affecting VLM performance, revealing key bottlenecks and directions for future improvement. 1 Introduction Recent work on AI agents has increasingly focused on building systems that can autonomously perceive, reason about, and act within user interfaces (UI) to complete complex tasks on users’ behalf ( 27 ; 17 ; 57 ; 30 ; 49 ; 55 ) . In real-world settings, such agents must also develop a rich understanding of user interfaces, including the ways in which interfaces convey system state, provide feedback, and signal available interaction affordances to users. A central yet underexplored aspect of this understanding is UI animation. Animation plays a fundamental role in modern user interface design to convey feedback and information 8 ; 46 ; 48 ; 21 ; 12 . These short yet critical animations serve more than aesthetic or experiential purposes; they are often essential for interpreting both the interface and the user’s interaction with it. For example, the MacOS dock icon bounces for notifications, and the password input box shakes on wrong password. In many cases, such animations are the primary or only channel through which this information is communicated. Unlike icons or illustrations, animation is defined more by movements that are drawn than by drawings that move 3 . Because animation’s meaning is typically encoded in motion rather than accompanying graphics, a still image alone is often insufficient to capture its intended message. Thus, a comprehensive UI understanding must account for both static content and dynamic animations. Figure 1: Overview of AniMINT, a UI animation dataset with multi-level human annotations. Each clip includes contextual information, an animation purpose label (highlighted in yellow), and ten annotations of the animation’s meaning and effect, supporting evaluations of VLMs across perception, purpose categorization, and interpretation. In this work, we evaluate the capabilities of state-of-the-art VLMs to understand UI animations. Recent VLMs have shown strong performance on a range of user interface understanding tasks and have been applied to increasingly complex UI-centered problems 44 ; 51 ; 38 . However, to the best of our knowledge, no prior works have systematically studied their capabilities to understand UI animations. To this end, we constructed AniMINT , the first UI AniM ation INT erpretation dataset. It contains 300 UI animation videos sourced from web, mobile, and desktop platforms. Animations were carefully annotated by 3 UI/UX practitioners and by 300 diverse users, providing a complementary view of UI animations from both experts and general users. We release our dataset and annotations 1 1 1 https://github.com/publicationacc/AniMINT . To systematically evaluate VLMs’ ability to understand UI animations, we formulate a set of research questions based on AniMINT that span both low-level animation recognition and higher-level animation understanding. We evaluated various state-of-the-art VLMs, including models from GPT, Gemini, and other model families. Our evaluation shows that, although most VLMs reliably recognize primitive animation effects, they struggle with higher-level purpose categorization and meaning interpretation. To diagnose further, we investigate how Motion, Context, and Perceptual Cues (MCPC) affect UI animation understanding. We augment inputs with motion blending, interaction context, and perceptual captions, then re-evaluate categorization and interpretation. The findings reveal the bottleneck in motion perception, while also highlight the importance of grounding motion in interaction context and higher-level semantic meaning for accurate interpretation. To summarize, our primary contributions are: • We introduce AniMINT , the first dataset for UI animation understanding, with diverse annotations from both experts and everyday users. • Using AniMINT, we conduct a systematic evaluation of nine state-of-the-art VLMs on both primitive animation perception and high-level animation categorization and understanding, revealing substantial gaps in current models’ capabilities. • We investigate factors that improve VLMs’ capabilities on UI animation understanding and show their effectiveness on Gemini-2.5-Flash. Figure 2: Dataset statistics. (Left) Distribution across seven animation purposes based on prior taxonomy. (Right) Animation duration by platform (mobile, web, and desktop). The median duration is 3.59s. 2 Related Work UI Animation. Based on 3 ; 6 ; 12 , we define UI animation as follows to guide data collection for AniMINT: UI animation is a deliberately constructed, dynamic transformation of a user interface element that visualizes information or evokes a perceptual or cognitive response in the user. The transformation extends beyond the immediate next frame. UI animations serve functional roles within the interface 46 ; 8 ; 29 , such as clarifying state transitions 18 ; 43 , visualizing information 48 ; 18 ; 43 , and enhancing user comprehension and experiences 34 ; 46 . Drawing from prior literature 3 ; 12 ; 37 ; 2 , we categorize animation purposes as Transition, Demonstration, Guidance, Feedback, Visualization, Highlight, and Aesthetic. We also categorize animation motion effects from the prior literature 46 ; 37 into 7 primitive effects, including move, rotate, size, color, fade, blur, and morph. These categorizations guide our data collection and annotation process, and detailed definition can be found in Appendix D . UI Animation Datasets. There have been datasets that include UI interaction recordings for UI understanding and computer use agent training, such as Rico 14 , MONDAY 25 , GUI World 9 , and others across different platforms and tasks 40 ; 56 ; 32 . To our knowledge, there is no dataset that specifically focuses on UI animation understanding. Existing datasets are typically designed for specialized tasks such as evaluating state transitions, UI adaptability, or visual signifiers for interaction discoverability 31 ; 18 . Although recordings may include animation, they lack the diversity and annotation needed to evaluate animation understanding. In contrast, AniMINT sources diverse UI animation videos annotated by 3 domain experts and 300 general users. VLMs and VLM Agent in UI Understanding. VLMs have emerged as powerful tools across various multimodal tasks, including visual scene comprehension, image captioning, and instructional task execution 19 ; 4 . More recently, VLM-based agents have extended these capabilities to interactive settings, enabling models to perceive, reason about, and act within complex visual environments by iteratively grounding language instructions in visual observations 53 ; 52 . Despite this, their application to UI understanding, particularly regarding dynamic animations, is less explored. Existing studies primarily focus on static UI elements, such as visual component identification, interface semantics extraction, and static screen analysis, rather