Overreliance in Writing Tasks: Exploring Similarity-Based Measures of AI Influence on Writing and Proposing a Reflective Writing Interface Intervention \acmConference@name ( \acmConference@shortname ), \acmConference@date , \acmConference@venue Conference: The 2026 ACM Conference on Fairness, Accountability, and Transparency; June 25–28, 2026; Montreal, QC, Canada The 2026 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’26), June 25–28, 2026, Montreal, QC, Canada DOI: 10.1145/3805689.3812389 ISBN: 979-8-4007-2596-8/2026/06 CCS: Human-centered computing HCI design and evaluation methods CCS: Computing methodologies Artificial intelligence CCS: Human-centered computing Empirical studies in HCI

Vitor H. A. Welzel, Nicholas Vincent · 2026

Read Overreliance in Writing Tasks: Exploring Similarity-Based Measures of AI Influence on Writing and Proposing a Reflective Writing Interface Intervention \acmConference@name ( \acmConference@shortname ), \acmConference@date , \acmConference@venue Conference: The 2026 ACM Conference on Fairness, Accountability, and Transparency; June 25–28, 2026; Montreal, QC, Canada The 2026 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’26), June 25–28, 2026, Montreal, QC, Canada DOI: 10.1145/3805689.3812389 ISBN: 979-8-4007-2596-8/2026/06 CCS: Human-centered computing HCI design and evaluation methods CCS: Computing methodologies Artificial intelligence CCS: Human-centered computing Empirical studies in HCI by Vitor H. A. Welzel, Nicholas Vincent (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15322.txt.

Introduction

Vitor H. A. Welzel and Nicholas Vincent study over-trust in writing with generative AI assistance. Forty-seven participants did analysis and combining tasks with or without AI help, and textual overlap between suggestions and final writing was measured with writer notes. AI help tracks patterns of suggestion reuse. A follow-up think-aloud study with four writers suggests a thoughtful writing interface can raise awareness of how AI output enters one's writing and support more conscious engagement with help.

Catalog record

CREATORVitor H. A. Welzel, Nicholas Vincent
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54230
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.15322

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Overreliance in Writing Tasks: Exploring Similarity-Based Measures of AI Influence on Writing and Proposing a Reflective Writing Interface Intervention \acmConference@name ( \acmConference@shortname ), \acmConference@date , \acmConference@venue Conference: The 2026 ACM Conference on Fairness, Accountability, and Transparency; June 25–28, 2026; Montreal, QC, Canada The 2026 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’26), June 25–28, 2026, Montreal, QC, Canada DOI: 10.1145/3805689.3812389 ISBN: 979-8-4007-2596-8/2026/06 CCS: Human-centered computing HCI design and evaluation methods CCS: Computing methodologies Artificial intelligence CCS: Human-centered computing Empirical studies in HCI Vitor H. A. Welzel, Nicholas Vincent Abstract . As generative AI (GenAI) systems become increasingly proficient at simulating human-like and well-reasoned text, users may attribute authority to AI outputs, shaping how they engage with writing and reasoning tasks. While prior work has raised concerns about AI overreliance , empirical approaches for observing this phenomenon during open-ended writing remain limited. In this paper, we examine how GenAI assistance influences users’ interactions with AI suggestions during writing. We report results from a mixed-methods study in which 47 participants completed analysis and synthesis writing tasks with or without AI assistance. We quantify the textual overlap between AI suggestions and participants’ writing and analyze participants’ reflections. Our results show that AI assistance is associated with patterns of suggestion reuse. Building on these findings, we design and evaluate an interactive writing interface that may support reflection on the usage of the AI suggestions during writing. Evidence from a small follow-up think-aloud study ( n = 4 n=4 ) suggests that the interface can increase users’ awareness of how AI outputs are incorporated into their writing and may support more conscious engagement with AI assistance. Together, our findings contribute empirical methods for studying AI adoption in writing contexts and demonstrate how interface design can shape user–AI interaction. † † cc-license: by 1. Introduction Generative AI (GenAI) systems are increasingly capable of producing text that reflects human reasoning cues, including a consistent natural tone and textual structure. Users may interpret these cues as indicators of credibility and expertise. Particularly in low - trust environments, people rely on these mental proofs to assess credibility and sincerity ( 46 ) . Previous work has shown that AI systems can simulate these proofs by crafting convincing arguments that users mistake for deep cognitive depth, therefore amplifying perceived expertise and trust ( 10 ) . Understanding how these perceptions shape user interaction with AI is an important challenge for human-computer interaction research. A growing research area highlights the tendency of overreliance on AI suggestions, accepting them even when they are incorrect, which can lead to poorer decision outcomes than independent reasoning ( 44 ; 7 ; 27 ; 5 ) . Interface interventions, such as explainable AI ( 44 ; 48 ) , cognitive forcing functions ( 7 ) , and clickable source references ( 27 ) , have shown some success in reducing overreliance, but often at the cost of usability or user satisfaction ( 7 ) . Much of the previous research has focused on binary decision-making tasks and has overlooked the impact of AI on open-ended tasks. Open-ended tasks generally require analysis and synthesis skills that are vital for human cognitive development. Unlike binary decision-making, which often yields clear answers, open-ended tasks require deeper thinking and understanding, resulting in more nuanced responses. They involve interpreting various factors, such as creating solutions that may not be straightforward. This nuance is crucial for developing higher-order cognitive skills, according to Bloom’s taxonomy ( 6 ; 21 ; 38 ) . Frequent usage of AI has also been linked to cognitive offloading, shifting users from "doing" to "overseeing" the AI work ( 29 ) as well as weakening neural indicators of mental engagement ( 28 ) . This shift may reduce opportunities to develop higher-order skills ( 29 ; 28 ; 8 ; 33 ) . Through an experiment, we investigate how users incorporate AI suggestions into their responses to open-ended tasks and further analyze their behaviour after completing each task. Building on the idea of overreliance , we provide additional evidence that people likely experience some degree of AI influence, i.e. the adoption of wording, structure, and sentiment. This is not necessarily a concern in some contexts — perhaps producing outputs that are similar to high-quality AI outputs is beneficial. Specifically, in a small experiment, we measure the effects of AI suggestions on textual similarity, sentiment alignment, and perceived cognitive load. Our findings illustrate the transition from more noticeable to more subtle yet meaningful ways in which generative AI can influence users’ responses. In general, we might expect that as people see AI suggestions, their outputs become more similar to the AI outputs. In some cases, this may be the user’s intention. In other cases, it might be cause for concern for the user or other people affected by the content. In a writing context, even seeing an AI suggestion might warp a creator’s eventual output. Therefore, the open question, and our focus, is what happens when users do not copy verbatim: to what extent do suggestions still imprint wording, structure, and sentiment on open-ended writing. To answer this, we argue that a single metric is insufficient; influence is not monolithic, and can manifest itself as lexical, structural, or tonal alignment. In this paper, we use the term adoption to refer to the incorporation of AI output into a participant’s written response , rather than to the general adoption of AI tools (e.g., increased ChatGPT usage). In practical terms, adoption refers to the extent to which a participant’s final text aligns with the provided suggestion in wording, structure, meaning, and sentiment. We use overreliance to refer to the broader concern from prior work. We discuss the main differences and when adoption becomes a concern in the Discussion section. We find that participants exposed to the fully formed AI suggestion exhibit higher levels of AI adoption . This adoption is reflected in the increase in textual and sentiment similarity metrics. Building on these findings, we designed and implemented an interactive writing interface that displays real-time metrics to users on AI response adoption. We evaluate this interface through a follow-up, exploratory think-aloud study, which shows that reflection on such metrics can increase users’ awareness of suggestion reuse. Our work produces evidence and design practices for LLM tools: • We conducted an empirical study with participants to assess whether viewing AI output leads users to produce more similar output. • We identified similarity metrics that may be particularly affected by AI assistance. • We design and evaluate an interface intervention that can promote reflective engagement with AI responses in writing workflows. 2. Related Work 2.1. Mental Proofs 46 describe mental proofs as a set of behaviours and cues that enable people to externalize their internal cognitive states ( 42 ) , such as understanding, sincerity and intention ( 18 ) . In low-trust environments, people often rely on these signals to assess others’ credibility ( 16 ) . For example, when reading news articles, we must rely on cues such as author credibility and presentation quality rather than directly verifying the facts. GenAI systems can simulate these mental cues by constructing convincing, well-structured arguments that lead users to mistake the AI’s

Read the complete text