Catch Me If You Can? Not Yet: LLMs Still Struggle to Imitate the Implicit Writing Styles of Everyday Authors

Zhengxiang Wang Nafis Irtiza Tripto Solha Park Zhenzhen Li Jiawei Zhou · 2025

Read Catch Me If You Can? Not Yet: LLMs Still Struggle to Imitate the Implicit Writing Styles of Everyday Authors by Zhengxiang Wang Nafis Irtiza Tripto Solha Park Zhenzhen Li Jiawei Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-14543.txt.

Introduction

Zhengxiang Wang and four co-authors test in 2025 whether large language models can imitate the personal writing style of everyday authors from a few samples. Personal style is subtle and implicit, hard to state in a prompt. Writing assistants still promise to sound like the user.

Their evaluation uses several complementary metrics: authorship attribution, verification, style matching and AI detection. Tens of thousands of generations per model were scored across news, email, forums and other domains. Models captured surface traits better than deep habits, the results suggest. Faithful imitation of implicit style remains out of reach for the systems tested. Assistants that claim to write exactly like you are, for now, overselling what few shot prompting delivers.

Catalog record

CREATORZhengxiang Wang Nafis Irtiza Tripto Solha Park Zhenzhen Li Jiawei Zhou
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-14543

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Catch Me If You Can? Not Yet: LLMs Still Struggle to Imitate the Implicit Writing Styles of Everyday Authors Zhengxiang Wang Nafis Irtiza Tripto Solha Park Zhenzhen Li Jiawei Zhou Abstract As large language models (LLMs) become increasingly integrated into personal writing tools, a critical question arises: can LLMs faithfully imitate an individual’s writing style from just a few examples? Personal style is often subtle and implicit, making it difficult to specify through prompts yet essential for user-aligned generation. This work presents a comprehensive evaluation of state-of-the-art LLMs’ ability to mimic personal writing styles via in-context learning from a small number of user-authored samples. We introduce an ensemble of complementary metrics—including authorship attribution, authorship verification, style matching, and AI detection—to robustly assess style imitation. Our evaluation spans over 40,000 generations per model across domains such as news, email, forums, and blogs, covering writing samples from more than 400 real-world authors. Results show that while LLMs can approximate user styles in structured formats like news and email, they struggle with nuanced, informal writing in blogs and forums. Further analysis on various prompting strategies such as number of demonstrations reveal key limitations in effective personalization. Our findings highlight a fundamental gap in personalized LLM adaptation and the need for improved techniques to support implicit, style-consistent generation. To aid future research and for reproducibility, we open-source our data and code. 1 1 1 https://github.com/jaaack-wang/llms-implicit-writing-styles-imitation . 1 Introduction Every time humans write, they leave behind a unique linguistic fingerprint Svartvik 1968 , a subtle and subconscious reflection of their personal style. To enhance their writing, humans have been using writing tools from spell and grammar checkers to paraphrasers for quite some time Ferris 2004 . But, the rise of large language models (LLMs) has introduced a powerful, all-in-one assistant capable of drafting, editing, and rephrasing text Wasi et al. 2024 . Despite their impressive fluency, LLMs often default to a generic style learned from vast web data, stripping away the personal touch that makes writing feel authentic Padmakumar and He 2024 . This raises concerns about authorship dilution and being flagged as AI-generated content Tripto et al. 2024 . Figure 1: Task of implicit personalized writing imitation: given few-shot writing examples from an everyday author and a content summary, an LLM is prompted to generate a novel text that imitates the writing examples in styles and reflect the semantics of the summary. To counter these concerns, some systems (e.g., Claude AI 2 2 2 https://claude.ai/ , accessed May 2025. ) offer configurable parameters like tone, voice, and formality. Yet these controls fall short as users’ personal styles are nuanced and rarely reducible to a few sliders Kang and Hovy 2021 . A more natural solution is to guide the model with examples of an individual’s prior writing, employing few-shot prompting to emulate an individual’s style more faithfully, like the task shown in Figure 1 . While LLMs have shown promise in mimicking public figures Herbold et al. 2024 ; Chen and Moscholios 2024 and fictional characters Li et al. 2023 whose extensive data footprints on the internet make them easy to model, their ability to replicate the style of everyday users remains largely unexplored. Can LLMs truly adapt to the personal style of an everyday author with only a handful of casual interactions and no explicitly defined stylistic identities? Existing research on personal-style generation for ordinary users is limited Bhandarkar et al. 2024 ; Cho et al. 2025 , and even more pressing is the lack of robust evaluation: how do we determine if a generated text from an LLM can belong to the same individual in style and voice? While the concept of authorship is inherently subjective,computational methods like Authorship Attribution (AA) and Authorship Verification (AV)—well-established in forensic linguistics—offer promising proxies. The theory of Linguistic Individuality Nini 2023 suggests that a person’s writing style often forms a consistent style model. If LLMs can generate text that fits within this style model while being recognized as from the same author by the two computational authorship models, it brings us closer to achieving truly personalized text generation. In this work, we examine whether LLMs can imitate the implicit writing styles of everyday users through in-context learning, without access to explicit stylistic instructions, via the generation task shown in Figure 1 . Our investigation is motivated by and rooted in a realistic usage scenario: most users interact with LLMs via standard interfaces rather than custom plug-ins or fine-tuned pipelines Wang et al. 2024 . While methods like personalized fine-tuning ( Liu et al. 2024 ) or advanced prompting strategies ( Cho et al. 2025 ) can improve stylistic alignment, they are often impractical in time-sensitive, real-world settings. Instead, we focus on evaluating the out-of-the-box ability of current LLMs to replicate personal writing style by conditioning on a few user-authored samples, reflecting the kind of implicit personalization that is most accessible and scalable in practice. To this end, we conduct a comprehensive evaluation of state-of-the-art LLMs on the task of personalized writing imitation , defined as emulating an individual’s implicit writing style solely from prior writing samples, without explicit stylistic instructions and semantically conditioned on a content summary (see Figure 1 ). Given the inherent difficulty of measuring personal style, we adopt a diverse suite of computational evaluators, including authorship attribution, authorship verification, linguistically grounded style metrics, and state-of-the-art AI detection tool. This ensemble approach enables a robust and multifaceted assessment of stylistic fidelity and human-like generation. Using this framework, we evaluate frontier models from leading providers, including OpenAI, Google, Meta, and DeepSeek, across writing domains such as news articles, emails, online forums, and personal blogs. These domains reflect a range of everyday writing scenarios with distinct stylistic constraints. Our datasets span over 400 authors and build on prior work to ensure diversity and realism. We investigate whether few-shot demonstrations of a user’s prior writing are sufficient for LLMs to generate outputs that match the author’s voice and style under our evaluators. Results show that while LLMs can partially emulate user style in more structured formats like news and email, they struggle with nuanced, informal expression in domains such as blogs and forums. Generated outputs often default to an average, generic tone and remain readily detectable as AI-written. Moreover, increasing the number of demonstrations offers limited gains in stylistic alignment. These findings underscore the current limitations of in-context personalization and highlight the need for more effective methods to achieve truly personalized generation, an essential step toward democratizing LLM-based writing tools. 2 Related Work Personalization in LLMs spans a wide spectrum, from persona emulation ( Tseng et al. 2024 ) to personalized content recommendations ( Lyu et al. 2024 ) . In this study, however, we specifically focus on the subtle challenge of implicit personalized writing imitation , whether LLMs can generate text that reflects the inherent implicit style learned from an individual’s prior writings. To situate our work, we review progress at the intersection of LLM-based personalized writing and authorship analysis around LLM generated texts. Figure 2: Methodology of our study. (A) We partition each author’s writing samples into training and test sets. We select few-shot examples from training samples to guide an LLM to generate novel

Read the complete text