IntentScore: Intent-Conditioned Action Evaluation for Computer-Use Agents
Rongqian Chen, Yu Li, Zeyu Fang, Sizhe Tang, Weidong Cao, Tian Lan · 2026
Read IntentScore: Intent-Conditioned Action Evaluation for Computer-Use Agents by Rongqian Chen, Yu Li, Zeyu Fang, Sizhe Tang, Weidong Cao, Tian Lan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-05157.txt.
Introduction
Chen and five colleagues propose IntentScore, a reward model that scores actions for computer-use agents. Such agents use large language models to run desktop tasks without judging action quality, so errors cascade. IntentScore learns from 398K offline GUI steps, using contrastive alignment for relevance and margin ranking for correctness. It embeds each planning intent, telling apart similar actions with different rationales. On tests it reaches 97.5 percent accuracy, and reranking Agent S3 on unseen OSWorld it lifts task success by 6.9 points.
Catalog record
| CREATOR | Rongqian Chen, Yu Li, Zeyu Fang, Sizhe Tang, Weidong Cao, Tian Lan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51313 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.05157
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
IntentScore: Intent-Conditioned Action Evaluation for Computer-Use Agents Rongqian Chen, Yu Li, Zeyu Fang, Sizhe Tang, Weidong Cao, Tian Lan Abstract Computer-Use Agents (CUAs) leverage large language models to execute GUI operations on desktop environments, yet they generate actions without evaluating action quality, leading to irreversible errors that cascade through subsequent steps. We propose IntentScore , a plan-aware reward model that learns to score candidate actions from 398K offline GUI interaction steps spanning three operating systems. IntentScore trains with two complementary objectives: contrastive alignment for state-action relevance and margin ranking for action correctness. Architecturally, it embeds each candidate’s planning intent in the action encoder, enabling discrimination between candidates with similar actions but different rationales. IntentScore achieves 97.5% pairwise discrimination accuracy on held-out evaluation. Deployed as a re-ranker for Agent S3 on OSWorld, an environment entirely unseen during training, IntentScore improves task success rate by 6.9 points, demonstrating that reward estimation learned from heterogeneous offline trajectories generalizes to unseen agents and task distributions. 1 Introduction Computer-Use Agents (CUAs) leverage large language models (LLMs) to plan and execute graphical user interface (GUI) operations on desktop environments. At each step, the agent observes the current screen (typically as a screenshot), reasons about the task progress, and generates executable GUI actions such as mouse clicks, keyboard inputs, and scrolls 22 ; 37 ; 1 ; 2 . Recent systems such as 10 ; 42 have demonstrated impressive capabilities on complex multi-step tasks spanning web browsing, document editing, and system administration. However, these systems share a critical limitation: a single LLM call produces a single action or simply selects the first or most frequent one, discarding the opportunity to compare alternatives 30 ; 29 . Recent studies show that CUAs frequently produce errors due to misalignment with the task context or intent 30 . These errors may cascade since the environment state shifts and recovery is difficult: once a wrong button is clicked or a dialog is dismissed, the environment state changes and recovery is difficult 16 . A natural mitigation is to sample multiple candidate actions, evaluate, rank and select the best one 33 . However, using the same LLM to self-evaluate its own candidates doubles inference cost and introduces self-preference bias 36 ; 23 , while methods that require parallel environment instances for test-time scaling are impractical under real-world compute and latency constraints. What is needed is a lightweight, independently trained model that can reliably distinguish correct actions from plausible-but-wrong alternatives without additional LLM calls or environment copies. We observe that large-scale offline GUI trajectory datasets already provide the raw material to build such a model. Datasets like AgentNet ( 13 ) contain hundreds of thousands of interaction steps across multiple operating systems, each annotated with per-step correctness labels. Crucially, different operating systems (Windows, Mac, Ubuntu) instantiate distinct MDPs that share abstract GUI patterns (dialog flows, menu hierarchies, form layouts) yet differ in visual appearance and interaction conventions. A reward model trained on this multi-environment data should learn patterns of action correctness that transfer across environment boundaries ( 12 ; 39 ) . We formalize this intuition through the lens of in-context reinforcement learning (ICRL) ( 50 ) : rather than learning a policy, we learn a reward estimator from offline trajectories spanning a diverse family of MDPs, which generalizes to unseen environments and agents at deployment time without further training. We propose IntentScore , a lightweight plan-aware dual-encoder reward model that instantiates this framework. IntentScore is pretrained on 398K GUI interaction steps across three operating systems and finetuned on the target Ubuntu domain, learning two complementary objectives: contrastive alignment (InfoNCE), which captures state-action relevance, and reward learning (margin ranking), which captures action correctness. A key finding is that contrastive alignment alone assigns nearly identical scores to correct and incorrect actions, and reward learning is essential to inject the correctness signal. Architecturally, IntentScore embeds each candidate’s planning intent in the action encoder rather than the state encoder, enabling discrimination between candidates with similar surface actions but different rationales. At deployment, IntentScore serves as a reward-guided re-ranker: it scores each candidate action generated by the CUA backbone and selects the highest-quality one, requiring no additional LLM calls, environment copies, or agent modifications. Our contributions are: • A plan-aware dual-encoder reward model that embeds each candidate’s planning intent in the action encoder, enabling discrimination between candidates with similar surface actions but different rationales. • A dual-objective training framework combining contrastive alignment (InfoNCE) and reward learning (margin ranking), addressing the finding that alignment alone captures state-action relevance but not action correctness (score gap 0.002 vs. 0.046). • A two-stage training pipeline (cross-OS pretraining on 398K steps across three OS families followed by target-domain finetuning) that yields +2.6 points on Ubuntu evaluation through cross-environment transfer. • End-to-end deployment validation: IntentScore integrated as a re-ranker for CUA baseline on OSWorld achieves 52.1% task success rate (+6.9 points), demonstrating that reward estimation learned from offline multi-environment data transfers to an unseen agent and task distribution. 2 Related work Computer-Use Agents. LLM-based GUI automation has progressed rapidly along two axes: architecture and data scaling , and action selection quality . On the first axis, systems have improved by scaling training data ( 5 ; 35 ) , introducing large-scale annotated trajectory datasets ( 37 ) , and designing specialized agent-computer interfaces ( 41 ) . Skill-based augmentation ( 38 ) and semantic-action bridging ( 21 ) further extend agent capabilities. On benchmarks such as OSWorld ( 40 ) , WebArena ( 49 ) , and AndroidWorld ( 28 ) , end-to-end success rates remain well below human performance, with failures frequently cascading from local action errors ( 48 ; 30 ) . On the second axis, recent work has begun to address action selection quality. Agent S3 ( 10 ) samples multiple candidates and selects via Best-of-N. GTA-1 ( 42 ) applies test-time scaling by sampling multiple action proposals and using an MLLM judge to select the best one, while Agent Alpha ( 33 ) introduces tree search over candidate trajectories. WMA Web Agent ( 4 ) takes a world-model approach, predicting the next state for each candidate via a learned transition model before scoring with a value function. Computer-Using World Model ( 11 ) further explores this direction by employing a diffusion model to simulate future screen states, enabling action evaluation through predicted visual outcomes. These methods improve action selection but rely on heavyweight components: additional LLM judge calls (GTA-1), environment rollouts (Agent Alpha), or a separate transition/diffusion model (WMA, CU-World Model). IntentScore instead trains a single lightweight reward model (13M parameters) that directly estimates action quality from (state, action, intent), requiring no additional LLM calls, environment copies, or transition model, while generalizing across environment boundaries via multi-OS offline training. In-context reinforcement learning. In-context reinforcement learning (ICRL) studies how agents can internalize the ability to adapt to unseen environments from offline trajectories, enabling zero-shot generalization without per-task gradient updates ( 50 ; 20 ) . Algorithm Distillation ( 14 ) distills RL learning histories into a Transformer’s forward pass, while DPT ( 15 ) pretrains on multi-task offline datasets to learn in-context decision-making. Subsequent work extends ICRL to partially observable settings ( 31 ) and broader task distributions ( 25