MindCopilot: Towards Formalizing and Evaluating Granular Human-LLM Co-Writing
Youqing Fang, Yinhao Tang, Yanan Sun, Jiangning Liu, Ziyi Wang, Xun Zhao, Bin Liu, Weiming Zhang, Kuikun Liu, Wenwei Zhang, Kai Chen · 2026
Read MindCopilot: Towards Formalizing and Evaluating Granular Human-LLM Co-Writing by Youqing Fang, Yinhao Tang, Yanan Sun, Jiangning Liu, Ziyi Wang, Xun Zhao, Bin Liu, Weiming Zhang, Kuikun Liu, Wenwei Zhang, Kai Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-23535.txt.
Introduction
Fang and colleagues formalize fine-grained human-LLM co-writing, where assistants suggest continuations unasked rather than waiting for prompts. Existing tests judge output quality only, missing how users accept, edit, or repair suggestions live, hiding true usability. The authors model writing as a human-in-the-loop Markov decision process and introduce the Co-Writing Fidelity Suite, metrics for user-assistant alignment and cognitive editing effort. A simulation across 16 writing domains with 1688 continuation queries, plus a 30-person study, shows interaction-aware tests reveal effects output-only metrics miss.
Catalog record
| CREATOR | Youqing Fang, Yinhao Tang, Yanan Sun, Jiangning Liu, Ziyi Wang, Xun Zhao, Bin Liu, Weiming Zhang, Kuikun Liu, Wenwei Zhang, Kai Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 104458 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.23535
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MindCopilot: Towards Formalizing and Evaluating Granular Human-LLM Co-Writing Youqing Fang, Yinhao Tang, Yanan Sun, Jiangning Liu, Ziyi Wang, Xun Zhao, Bin Liu, Weiming Zhang, Kuikun Liu, Wenwei Zhang, Kai Chen Abstract Recent writing assistants are increasingly shifting from passive, prompt-driven interaction to proactive, suggestion-based completion, which integrates localized continuations into the writing flow and reduces coordination burden. However, existing evaluations simply focus on output quality, failing to capture how users accept, edit, or repair suggestions in real-time interaction, and thus obscuring the true usability of proactive co-writing systems. To address this gap, we adopt a sequential, behavior-centered view of interactive writing and formalize co-writing as a Human-in-the-Loop Markov Decision Process, modeling writing as an interaction shaped by user acceptance and editing decisions. Based on this formulation, we introduce the Co-Writing Fidelity Suite, an interaction-aware metric suite that captures both user–assistant alignment and cognitive editing effort, including Hierarchical Acceptance Rate and Knowledge-aware Editing Distance. We conduct a large-scale simulation study across 16 writing domains, using 1,688 controlled continuation queries sampled from different writing stages. Our analysis reveals systematic effects of interaction structure on acceptance behavior and editing cost. A follow-up user study with 30 participants confirms that these behavioral patterns align with real user experience. Together, our findings demonstrate that interaction-aware evaluation provides insights beyond output-only metrics and informs the design of more effective proactive writing assistants. † † footnotetext: * Equal contribution, {\dagger} Corresponding author. † † footnotetext: https://github.com/fangyouqing/MindCopilot 1 Introduction Driven by rapid advances in Large Language Models (LLMs) Liu et al. 2025 ; Comanici et al. 2025 ; Singh et al. 2025 , digital writing workflows have been reshaped, enabling users to draft, revise, and expand text with unprecedented ease Anthropic 2025 ; OpenAI 2025 . In most existing systems, writing assistance is realized through prompt-driven interaction , where the model remains largely passive until explicitly invoked Bai et al. 2022 ; Pasch and Ha 2025 . Authors issue discrete requests, receive complete responses, and manually integrate them into an evolving document. While effective for content generation, this paradigm places the burden of initiative and coordination squarely on the user, offering limited support during the continuous flow of writing Reza et al. 2025 ; Mysore et al. 2025 ; Dhillon et al. 2024 ; Li et al. 2026 . Recent writing interfaces increasingly adopt a suggestion-based interactive completion paradigm Lee et al. 2022 ; Lee et al. 2022 ; Laban et al. 2024 , in which the assistant acts more proactively within the writing process. Rather than waiting for prompts, the system surfaces localized continuations directly in the text, and authors iteratively accept, revise, or reject them. This mode of collaboration preserves human agency while reframing writing as a recurrent decision process—one centered on suggestion review, selective acceptance, and post-hoc editing . Despite its growing practical relevance, how users behave in such proactive co-writing settings remains poorly understood. Figure 1: Two paradigms of LLM-assisted writing. Prompt-driven interaction interrupts writing flow and shifts coordination burden to the user (left), whereas suggestion-based interaction integrates proactive, localized completions into the writing process, sustaining flow and reducing post-hoc editing (right). This gap is mirrored in prevailing evaluation practices for writing assistants. Widely used paradigms—such as LLM-as-a-Judge or Arena-style comparisons—assess generated text largely in isolation Chung et al. 2025 ; Li et al. 2025 ; Liao et al. 2025 ; Ying et al. 2025 ; Fein et al. 2025 ; Jia et al. 2025 , abstracting away the human actions required to integrate suggestions into an ongoing document. As a result, generation quality is often conflated with usability: metrics capture how good a continuation looks, but not whether it aligns with author intent or meaningfully reduces editing effort. Such evaluations fall short for interactive writing, where acceptance behavior and editing cost are as central as surface-level text quality. To address this limitation, we formalize interactive co-writing as a Human-in-the-Loop Markov Decision Process (HitL-MDP) . In this formulation, writing is modeled as a sequential decision problem: the assistant generates context-aware suggestions, the user responds through accept/reject/edit actions, and these actions drive state transitions that shape future assistance. This abstraction provides a principled framework for analyzing how different interaction structures influence user behavior over time, while admitting a operationalization in which acceptance decisions can be instantiated as a learnable component for downstream optimization. Building on this framework, we introduce the Co-Writing Fidelity Suite , a set of interaction-aware metrics designed to evaluate human–AI collaboration along two complementary dimensions: alignment and interaction friction. First, we propose Hierarchical Acceptance Rate (HAR) , which treats acceptance not as a binary outcome but as a structured, multi-stage decision, reflecting the cascading filters users implicitly apply during suggestion review. Second, we introduce Knowledge-aware Editing Distance (KED) , which quantifies post-acceptance editing effort by weighting edits according to their cognitive cost, recognizing that correcting knowledge-dense content or factual errors demands substantially more effort than surface-level revisions. Using these metrics, we conduct a large-scale simulation study across multiple interaction paradigm. We curate a dataset of 60 human-authored articles spanning 16 writing domains and construct 1,688 controlled continuation queries sampled from different writing stages. By varying interaction structures while holding content constant, we analyze how factors such as interaction history affect acceptance behavior and editing effort. Guided by insights from these simulations, we further design an adaptive interaction policy and implement a real-world writing interface. A subsequent user study with 30 participants shows that behavioral patterns observed in simulation closely align with subjective user experience. Our contributions are threefold: • We cast interactive co-writing as a Human-in-the-Loop sequential decision process, providing a principled abstraction for modeling user acceptance, rejection, and editing behaviors over time. • We introduce the Co-Writing Fidelity Suite, a set of interaction-aware metrics—including Hierarchical Acceptance Rate and Cognitive-Weighted Editing Distance—that quantify user–assistant alignment and the cognitive effort required to integrate suggestions. • We conduct a large-scale empirical study combining controlled simulation and real-user experiments, demonstrating that interaction-aware evaluation based on the Co-Writing Fidelity Suite reveals behavioral patterns invisible to output-only metrics. 2 Related Work 2.1 Human-AI Co-Writing: From Tool to Collaborative Agency The co-writing paradigm is evolving from passive, auxiliary local completion Lee et al. 2022 ; Coenen et al. 2021 to mixed-initiative collaboration that frames AI as a “co-operative partner” preserving user ownership Ashkinaze et al. 2025 ; Zhang et al. 2025 ; He et al. 2025 ; Yang et al. 2022a . However, evaluation methodologies lag behind. Traditional metrics fail to capture dynamic user experience Shen and Wu 2023 , while recent attempts at quantifying effort Devatine and Abraham 2024 or keystroke intent Le et al. 2025 suffer from coarse granularity and rigid definitions. There remains a notable absence of fine-grained, entity-aware metrics to measure cognitive friction in complex interactions. 2.2 Long-form Writing: Control and Alignment With improved context capabilities, generation tasks now encompass document-level long-form content. While works employ outline-driven, recursive strategies to ensure long-range coherence Mirowski et al. 2023 ; Yang et al. 2022b ; Yang et al. 2023 ; Bai et al. 2024 —evaluated via coarse-grained dynamic metrics Du et al. 2025 ; Wu et al. 2025 ; Que et al. 2024 —they predominantly rely on “black-box” monolithic generation. This approach leads to “loss of