Learn from Weaknesses: Automated Domain Specialization for Small Computer-Use Agents

Suji Kim, Kangsan Kim, Sung Ju Hwang, [0.4em] KAIST · 2026

Read Learn from Weaknesses: Automated Domain Specialization for Small Computer-Use Agents by Suji Kim, Kangsan Kim, Sung Ju Hwang, [0.4em] KAIST (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28775.txt.

Introduction

Suji Kim, Kangsan Kim, and Sung Ju Hwang present LearnWeak, an annotation-free framework for small computer-use agents. Large experts per domain cost too much, and small open agents stay weaker with uneven domain failures, while naive training data gives only marginal gains. LearnWeak uses a reference agent to spot the student's weaknesses, synthesize tasks, and build supervision, with an error-aware objective separating planning and execution errors. On OSWorld it gains about 12 and 11 points over EvoCUA-8B and OpenCUA-7B across eight domains, beating autonomous trajectory generation baselines.

Catalog record

CREATORSuji Kim, Kangsan Kim, Sung Ju Hwang, [0.4em] KAIST
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78397
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.28775

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learn from Weaknesses: Automated Domain Specialization for Small Computer-Use Agents Suji Kim, Kangsan Kim, Sung Ju Hwang, [0.4em] KAIST Abstract Computer-use agents (CUAs) have recently made substantial progress, but deploying a separate large expert for each software domain remains expensive. Small open CUAs are more practical specialization targets, but they remain substantially weaker and exhibit uneven domain-specific failures. A straightforward remedy is to synthesize large-scale training data for the target domain, yet we find that this naive approach yields only marginal improvements. Building on this observation, we introduce LearnWeak , an annotation-free specialization framework for small CUAs that uses a stronger reference agent to identify the student’s weaknesses in the target domain, synthesize targeted tasks, and construct supervision automatically. LearnWeak further introduces an error-aware specialization objective that disentangles planning and execution errors, enabling more behaviorally precise updates than broad uniform supervision. On OSWorld, LearnWeak achieves average gains of 11.6 and 11.1 percentage points over EvoCUA-8B and OpenCUA-7B, respectively, across eight domains. We also validate that our student-aware dataset generation and training approaches outperform existing autonomous trajectory generation and training baselines. Our work highlights the importance of student-awareness in both data synthesis and agent training, pointing toward a more principled and efficient path for specializing small CUAs in diverse domains. 1 Introduction Computer-use agents (CUAs) have advanced rapidly across desktop and web environments, with two dominant paradigms emerging: large proprietary models such as Claude Sonnet 4.6 [ 2 ] and GPT-5.4 [ 24 ] , and small models fine-tuned specifically for computer-use tasks, such as EvoCUA [ 42 ] and OpenCUA [ 36 ] . The latter paradigm [ 37 , 45 , 25 ] is particularly compelling for real-world deployment, as fine-tuned small models enable faster and more cost-efficient inference while remaining viable for edge devices [ 40 , 18 ] and privacy-sensitive enterprises where proprietary APIs are prohibited [ 7 , 47 ] . However, a substantial performance gap persists between closed models and small CUAs, particularly in domain-specific software environments with unique conventions or unfamiliar workflows [ 16 , 44 , 39 ] . Addressing this gap is therefore critical for advancing the practical deployment of small CUAs. Figure 1 : Conceptual illustration of LearnWeak and performance gains after domain specialization, showing consistent improvements of the small student across target software domains. Domain specialization, which fine-tunes agents on a single target domain, is a promising approach for improving the performance of small CUAs. Small models often lack the capacity to simultaneously learn the workflows of diverse software environments, and training across heterogeneous computer-use tasks can lead to catastrophic forgetting and degraded performance within individual domains [ 13 , 12 , 20 ] . Although scaling up data or designing more sophisticated training objectives can help, both require significant annotation effort or computational cost [ 36 , 17 , 35 ] . By contrast, domain-specialized training can improve sample efficiency by focusing on domain-specific interaction patterns rather than broad generalization. Recent studies [ 32 , 19 , 31 , 5 ] provide empirical evidence supporting the effectiveness of this approach for small CUAs. Domain specialization for CUAs consists of two stages: dataset generation and agent training. In the dataset generation stage, collecting human trajectories is costly due to the long-horizon nature of computer-use tasks, which makes autonomous trajectory generation essential [ 43 , 38 ] . Existing fixed data generation strategies do not consider student deficiencies on the target domain, resulting in inefficient training [ 9 , 30 , 10 ] . However, data quality matters as much as data quantity: to specialize efficiently, generated queries should target model weaknesses and missing domain knowledge rather than reinforcing already well-learned skills. In the training stage, domain specialization must preserve pretrained agentic capabilities while selectively repairing weaknesses. Small CUAs develop their own reasoning patterns and recovery mechanisms, and naive fine-tuning can distort these by imposing human or large-model reasoning distributions that diverge from the agent’s own [ 15 , 21 ] . Moreover, failure modes are heterogeneous even within a single model: some failures stem from incorrect planning, whereas others arise from execution errors such as inaccurate coordinates [ 8 , 39 , 1 ] . These challenges call for a framework that identifies the student’s weaknesses in the target domain and applies tailored training objectives. To address these challenges, we introduce LearnWeak , a fully automated domain specialization framework for small CUAs that targets student weaknesses across both dataset generation and agent training. For the dataset generation stage, we propose an annotation-free pipeline that expands the training set through repeated cycles of teacher-student comparison, weakness analysis, and query synthesis. It requires only a small set of seed queries, yet produces a compact and targeted dataset that addresses the student’s deficiencies. For agent training, we introduce an error-aware preference optimization which adaptively targets task-specific weaknesses. It dynamically adjusts the training objective according to the failure type, distinguishing between planning and execution failures. Together, our student-aware data generation and training enable small CUAs to close capability gaps on the target domain without human annotation. We evaluate LearnWeak across 8 OSWorld domains [ 39 ] using EvoCUA-8B [ 42 ] and OpenCUA-7B [ 36 ] as base students. Our domain specialization improves average performance by 11.6 and 11.1 percentage points on EvoCUA-8B and OpenCUA-7B, respectively. Notably, the specialized small agents surpass the teacher on several domains, and our data-generation pipeline achieves the strongest gains among autonomous generation baselines under matched budgets. We further show that error-aware preference optimization outperforms alternative offline training strategies, including SFT and standard DPO variants. We hope this work serves as a foundation for more efficient and targeted domain specialization of small CUAs and encourages future research toward closing the performance gap between small open models and large proprietary agents. 2 Preliminaries 2.1 Computer-Use Agent A computer-use agent (CUA) is a policy that operates within an interactive software environment by perceiving the screen and issuing actions to complete a given task. Since the current screen alone does not reveal the full environment state, CUA settings are better modeled as a partially observable decision process (POMDP) [ 14 ] . Following common practice [ 48 , 42 , 6 ] , we handle this partial observability by conditioning the policy on the full interaction history. At each step t , the agent receives the current screen as a partial observation of the environment state together with the interaction history h_{t}=(o_{1},a_{1},o_{2},a_{2},\dots,o_{t-1},a_{t-1}) , which records all previously observed screens o_{<t} and executed actions a_{<t} . Conditioned on the current context c_{t}=(q,o_{t},h_{t}) where q is the task instruction, the agent policy \pi produces a structured output, a_{t}=\pi(c_{t})=(r_{t},\,s_{t},\,e_{t}). (1) It consists of three components: (i) internal reasoning r_{t} , which reflects the agent’s analysis of the current state; (ii) an action description s_{t} , a natural language description of the intended action; and (iii) tool execution e_{t}=(f_{t},p_{t}) , the executable action that directly manipulates the environment, consisting of a function type f_{t} and its parameters p_{t} such as left_click(x,y) or type(text) . The agent repeats this process until the task is complete, producing the full trajectory \tau=(o_{1},a_{1},o_{2},a_{2},\dots,o_{H},a_{H}). (2) 2.2 Problem Formulation We address domain specialization , namely domain-specific finetuning of a broadly capable student

Read the complete text