Small Models, Big Results: Achieving Superior Intent Extraction through Decomposition
Danielle Cohen, Yoni Halpern, Noam Kahlon, Joel Oren, Omri Berkovitch, Sapir Caduri, Ido Dagan, Anatoly Efros · 2025
Read Small Models, Big Results: Achieving Superior Intent Extraction through Decomposition by Danielle Cohen, Yoni Halpern, Noam Kahlon, Joel Oren, Omri Berkovitch, Sapir Caduri, Ido Dagan, Anatoly Efros (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-12423.txt.
Introduction
Danielle Cohen and 7 co-authors show how small models can extract user intent from screen interactions in a 2025 paper. Understanding what a user meant to do from a sequence of interface actions is important for intelligent agents. Large multimodal models handle such sequences well but need data centers, raising cost, delay, and privacy concerns.
Their decomposed approach first summarizes each interaction in a structured way, capturing key information from every action. A fine-tuned small model then extracts intent from those combined summaries. The method improved intent understanding in resource-constrained models that can run on a device. In their tests, it even surpassed the base performance of large multimodal models, suggesting that breaking a hard task into steps can beat raw scale.
Catalog record
| CREATOR | Danielle Cohen, Yoni Halpern, Noam Kahlon, Joel Oren, Omri Berkovitch, Sapir Caduri, Ido Dagan, Anatoly Efros |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-12423
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Small Models, Big Results: Achieving Superior Intent Extraction through Decomposition Danielle Cohen, Yoni Halpern, Noam Kahlon, Joel Oren, Omri Berkovitch, Sapir Caduri, Ido Dagan, Anatoly Efros Abstract Understanding user intents from UI interaction trajectories remains a challenging, yet crucial, frontier in intelligent agent development. While massive, datacenter-based, multi-modal large language models (MLLMs) possess greater capacity to handle the complexities of such sequences, smaller models which can run on-device to provide a privacy-preserving, low-cost, and low-latency user experience, struggle with accurate intent inference. We address these limitations by introducing a novel decomposed approach: first, we perform structured interaction summarization, capturing key information from each user action. Second, we perform intent extraction using a fine-tuned model operating on the aggregated summaries. This method improves intent understanding in resource-constrained models, even surpassing the base performance of large MLLMs. 1 Introduction Advancements in the capabilities of multi-modal large language models (MLLMs) has led to recent interest in modeling sequences of user interactions with phone and web graphical interfaces, both for the purposes of automation ( Wang et al. 2024b ; Jiménez-Ramírez 2024 ) , and understanding ( Berkovitch et al. 2025 ; Zhang et al. 2025 ) . In this work, we focus on the user intent extraction task, which consists of producing a free-form description of the inferred intent of a user from a sequence of interactions with a device. Large MLLMs are naturally fairly good at this task, however, it is more challenging for smaller models (E.g., Gemini 1.5 Flash 8B ( Gemini Team et al. 2024 ) or Qwen2 VL 7B ( Wang et al. 2024a ) ). The performance of smaller models is important for user interaction tasks due to their ability to operate within a private, on-device environment like a phone or browser, with reduced cost, energy usage, and latency ( Xu et al. 2024 ) . Figure 1: Proposed intent extraction flow (Described in detail in Section 4 ). Individual interactions (input as action strings and screenshots) are summarized and then the summaries are combined to output a short inferred intent for the trajectory. The summarization step uses both action strings and visual screen information to output a structured summary with two fields corresponding to screen summary (top, blue) and user action (bottom, red). In this paper, we introduce a two-stage approach for extracting user intent with small models. In the first stage, each atomic interaction is summarized. In the second stage, the full sequence of summaries is fed to a second model which outputs an intent. The overall flow is illustrated in Figure 1 . Using semantic equivalence metrics on public UI automation data, our two-stage approach demonstrates superior performance compared to both smaller models and a state-of-the-art large MLLM, independent of dataset and model type. Our approach also naturally handles scenarios with noisy data that traditional supervised fine-tuning methods struggle with. The modular nature of the architecture is helpful from an engineering perspective, allowing us to evaluate the approach in detail and identify key areas to improve. Our contributions can be summarized as follows: 1. We describe an effective decomposition of intent-extraction that unlocks the potential of small models; 2. We present non-trivial design components related to each stage of the decomposition; 3. We extensively evaluate our approach and demonstrate the effectiveness of our method across a range of data sets, base models and metrics. 2 Background 2.1 Intent Extraction from UI Interactions We formalize the intent extraction task, sometimes called goal understanding, similarly to prior works Berkovitch et al. 2025 and Zhang et al. 2025 . Consider a user journey T within a mobile or web application, represented as a sequence of interactions: T=(I_{1},I_{2},...,I_{n}) , where each interaction I_{i}=(O_{i},A_{i}) consists of an observation, O_{i} , and the action, A_{i} , the user performed at that step. This description is general and different modeling approaches have used different representations for observations and actions (e.g., textual descriptions, screenshot images, DOM hierarchies, etc.) ( Rawles et al. 2023 ; Burns et al. 2022 ) . The objective of the intent-extraction task is to generate a free-form sentence describing the user’s intent. Effectively, this setting can be thought of as the inverse problem of the UI automation task, with inputs and outputs swapped. Rather than producing a sequence of actions from an instruction, we ask “what was the user trying to accomplish with this trajectory?”. Intent extraction has been identified as an important building block for UI automation tasks, proactive assistance, and personalized memory ( Berkovitch et al. 2025 ; Zhang et al. 2025 ) . Very recently, a few works have begun addressing intent extraction from UI interactions. Berkovitch et al. 2025 proposed this novel task, and seemingly were the first to point out that it can be viewed as the inverse task of UI Automation. They evaluated MLLM intent-extraction performance over UI automation datasets (swapping input/output roles). As input, they considered screenshot images and textual descriptions of user actions, as we do in our work, and assessed the performance of standard MLLMs using a fairly simple prompting approach. In our work, we follow this evaluation approach, while testing an improved version of their prompt as a baseline. Zhang et al. 2025 also evaluated their SummAct model over UI automation datasets, but took as input only short textual descriptions of the specific UI elements with which the user interacted and the respective user actions, without considering the global screen context, as we do. Loosely similarly to our method, they decomposed intent generation into two consecutive stages, though these are substantially different than ours and required intervening with the LLMs attention mechanism for fine-tuning (see Appendix H for detailed comparison of the two methods). Finally, the UI-JEPA model ( Fu et al. 2025 ) takes as input videos of the entire UI session. Their method adapts a specialized video-embedding method to work with videos of UI sessions, and then finetunes an LLM decoder that generates intent descriptions based on these embeddings. Overall, SummAct, our work, and UI-JEPA consider different types of inputs, providing increasingly richer contexts with increasing complexity: ‘‘local’’ text descriptions of the user action and its respective UI element, accounting for the full screen image, and processing a complete video of the entire session (respectively). It is left for future work to thoroughly explore the pros and cons of these alternative inputs under different scenarios. 1 1 1 An empirical comparison with SummAct on equal grounds was not possible, since their code was released without full prompts close to the submission deadline. Similarly, the UI-JEPA inference code hasn’t been released yet while the license on the dataset restricts our lab’s usage. With respect to evaluating model-predicted intents, a good intent is faithful: only describes things that actually occur in the trajectory; comprehensive: provides all of the information about the user intent required to re-enact the trajectory; and relevant: does not contain extraneous information beyond what is needed for comprehensiveness. However, even with a well-defined ground truth intent, accurately evaluating a model’s extracted intent is challenging. User intents often contain many details, such as trip planning specifics or transaction data, which require metrics that can handle partial matches. Such metrics fall into two categories: semantic, which analyze underlying