VerificAgent : Integrating Expert Knowledge and Fact-Checked Memory for Robust Domain-Specific Task Planning

Thong Q. Nguyen, Shubhang Desai, Yash Jain, Tanvir Aumi, Vishal Chowdhary · 2025

Read VerificAgent : Integrating Expert Knowledge and Fact-Checked Memory for Robust Domain-Specific Task Planning by Thong Q. Nguyen, Shubhang Desai, Yash Jain, Tanvir Aumi, Vishal Chowdhary (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-02539.txt.

Introduction

Nguyen and co-authors present VerificAgent, a fact-checking layer for computer-using agents, in a 2025 paper. Agents that learn from stored memories of past tasks can pick up bad habits: shortcuts that clash with user intent or safety norms in a given domain. Unvetted memory becomes a quiet alignment risk.

The framework starts from an expert-written seed of domain knowledge. Memories then grow from real task runs during training. Before deployment, humans fact-check the stockpile and scrub the bad rules. On desktop productivity tasks and adversarial stress tests, the verified memory improved reliability and cut hallucination-driven failures, all without retraining the model. The memory acts as a frozen safety contract.

Catalog record

CREATORThong Q. Nguyen, Shubhang Desai, Yash Jain, Tanvir Aumi, Vishal Chowdhary
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-02539

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VerificAgent : Integrating Expert Knowledge and Fact-Checked Memory for Robust Domain-Specific Task Planning Thong Q. Nguyen, Shubhang Desai, Yash Jain, Tanvir Aumi, Vishal Chowdhary Abstract Continual memory augmentation allows computer-use agents (CUAs) to learn from past interactions and refine their task-solving strategies over time. However, unchecked memory accumulation can introduce spurious or hallucinated “learnings” that degrade agent performance—particularly in domain-specific workflows such as productivity software. We present a novel framework, VerificAgent , that effectively manages memory for CUAs through (1) an expert-curated seed of domain knowledge, (2) iterative, trajectory-based memory refinement during training, and (3) a post-hoc fact-checking pass by human experts to sanitize accumulated memory before deployment. On OSWorld productivity tasks, VerificAgent achieves a 111.1% relative improvement in success rate over baseline CUA without any additional fine-tuning. † † footnotetext: † Equal contribution. 1 Introduction Autonomous computer-using agents (CUAs) have the potential to transform how users interact with software by delegating routine digital tasks to AI. Recent advances in large language models (LLMs) with multimodal capabilities have enabled agents that can interpret and manipulate graphical user interfaces (GUIs) much like a human operator. For example, OpenAI’s Operator system is trained to read screen content and perform clicks, typing, and other GUI actions without relying on application-specific APIs [ 1 ] . This generality allows a single model to work across web and desktop applications. However, while such agents demonstrate impressive versatility, they still struggle with complex, domain-specific tasks. In particular, office productivity applications (e.g., Microsoft Word and PowerPoint) pose a significant challenge for current CUAs, which have been primarily trained on web browsing and simple tool use rather than the intricacies of Office software. Benchmark evaluations underscore this gap. OSWorld [ 2 ] provides a realistic operating system environment with tasks spanning real-world apps, and reports that state-of-the-art agents solve only about 12% of tasks overall, far below human performance ( \approx 72 %). Even with subsequent improvements—OpenAI’s CUA model reached 38% success on OSWorld’s full task suite—performance on Office-specific tasks remains around 25% success in the best models [ 3 , 1 ] . In practice, current agents often fail at fine-grained Office operations like precise text formatting or slide manipulation, which require remembering complex instructions and adapting to detailed interface layouts. This motivates the need for methods that can boost the task efficiency of CUAs in such complex domains. Recent work has shown that augmenting LLM agents with persistent memory can substantially improve multi‐step task adaptation and planning [ 4 , 5 ] . However, these methods do not guard against the gradual accumulation of spurious or hallucinated entries, which can mislead downstream decision‐making [ 6 ] . From a theoretical standpoint, verification is often easier than generation—checking a candidate solution requires fewer resources than constructing one—and this distinction carries over to LLMs: empirical studies find that models verify or critique outputs more reliably than they generate them from scratch [ 7 , 8 ] . Practical frameworks like LLM-Cite demonstrate that fact verification can be performed cheaply and efficiently by generating citation URLs and then validating claims against retrieved documents, reducing verification cost by over an order of magnitude compared to full generative pipelines [ 9 ] . Similarly, tools such as SymGen provide direct, in‐context citations that speed human validation of model outputs by presenting source evidence alongside responses [ 10 ] . Moreover, in domains where no structured knowledge base exists, a lightweight human review of distilled memory entries can still be carried out with minimal effort, ensuring reliability without major overhead. Despite these advances in rapid verification, memory‐based agents have largely overlooked the need to sanitize their learned memory stores. No prior system integrates a post-hoc fact-checking pass to prune or correct hallucinated memory entries before inference in a production environment, motivating our practical framework, VerificAgent , which combines expert‐seeded priors, iterative memory learning, and a targeted fact-checking stage to ensure robust, production-ready performance. Our key contributions are: • Scalable Continual Learning Framework. We propose a CUA training pipeline that combines a small, expert-curated domain knowledge base with iterative, trajectory-driven memory updates—allowing the agent to refine its planning strategy over successive trials. • Practical Memory Fact-Checking. We introduce a post-hoc verification step in which human experts review and correct the accumulated memory, removing hallucinated or incorrect entries to produce a robust, production-ready knowledge store. • Precision Office Grader Enhancements. We improve the OSWorld Office task grader by tightening comparison logic and addressing structural edge cases in PPTX files, yielding more accurate, function-level assessments of agent success. 2 Related Work 2.1 Computer-Using Agents and Benchmarks OpenAI introduced the Computer-Using Agent (CUA) paradigm with Operator, achieving a 38% success rate on the OSWorld benchmark [ 1 ] . OSWorld provides 369 real-world tasks—spanning web browsing, file management, and desktop application use—and highlights a large gap between human ( \approx 72 %) and agent performance (12 %) [ 2 ] . OfficeBench zeroes in on multi-step Office productivity workflows, reporting that even GPT-4-based agents plateau at around 47% success [ 3 ] . Additional execution benchmarks include DiGIRL for in-the-wild device control [ 11 ] and AndroidWorld for mobile GUI agents [ 12 ] , both underscoring challenges in generalization and error recovery. 2.2 GUI Grounding and Office Task Automation Vision-based grounding and screen parsing enable platform-agnostic GUI interaction. OmniParser converts screenshots into structured GUI elements to enhance LLM grounding, boosting performance on ScreenSpot and WebShop benchmarks [ 13 ] . UGround trains a universal visual-grounding model on a 10 M-element GUI dataset, supporting pixel-level operations across desktop, web, and mobile without relying on accessibility APIs [ 14 ] . WinClick advances Windows-specific grounding by combining screenshot-based detection with LLM alignment, achieving superior results on the WinSpot benchmark [ 15 ] . Unlike these perception-focused systems, our work targets the decision-making layer by integrating initial domain knowledge injection, trajectory-based memory, and post-hoc fact-checking to drive robust planning for Office-specific tasks. 2.3 Compositional and Modular Agent Architectures To tackle the complexity of long-horizon GUI tasks, recent works propose hierarchical and modular designs. Agent S introduces an experience-augmented hierarchical planner and an Agent–Computer Interface (ACI) for domain knowledge acquisition and dynamic interface handling, setting new SOTA on OSWorld and WindowsAgentArena [ 16 ] . Agent S2 extends this with a Compositional Generalist–Specialist framework—using a Mixture-of-Grounding mechanism for precise GUI localization and Proactive Hierarchical Planning—outperforming monolithic agents across OSWorld, WindowsAgentArena, and AndroidWorld benchmarks [ 17 ] . UI-TARS pioneers a native end-to-end GUI agent that learns from iterative reflective traces and vision grounding, achieving 24.6% success on OSWorld (50-step) and 46.6% on AndroidWorld using only screenshots as input [ 18 ] . 2.4 Planning and Memory in LLM Agents 2.4.1 Planning Interleaving reasoning with action has become a cornerstone of LLM‐driven agents: ReAct combines chain‐of‐thought traces with environment actions to reduce hallucinations in single episodes [ 19 ] , and Reflexion adds a self‐reflection loop that stores textual “reflections” as episodic memory for future planning [ 20 ] . Retrieval‐Augmented Generation (RAG) further boosts plan reliability by injecting external context—surveys of RAG for meeting‐scale LLMs [ 21 ] and contextual memory mechanisms like RAP [ 22 ] demonstrate substantial gains. Tool‐use frameworks such as Toolformer [ 23 ] ,

Read the complete text