Evaluating Style-Personalized Text Generation: Challenges and Directions
Anubhav Jangra, Bahareh Sarrafzadeh, Adrian de Wynter, Silviu Cucerzan, Sujay Kumar Jauhar · 2025
Read Evaluating Style-Personalized Text Generation: Challenges and Directions by Anubhav Jangra, Bahareh Sarrafzadeh, Adrian de Wynter, Silviu Cucerzan, Sujay Kumar Jauhar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06374.txt.
Introduction
Anubhav Jangra and co-authors examine how the field judges AI that writes in a person's own style, in a 2025 paper. Style-matched writing is a fast-growing use of language models. Style is personal and depends on the setting, which makes it hard to score. Past benchmarks and metrics are not standard and often track human judgment poorly.
The authors put the common metrics on trial. Word-overlap scores, embedding scores, and model judges were all tested. Their new benchmark asks whether a metric can tell one writer's style from another. It spans eight writing tasks. If the metrics fail that test, the field may overrate its systems. The paper points to where better scoring should go.
Catalog record
| CREATOR | Anubhav Jangra, Bahareh Sarrafzadeh, Adrian de Wynter, Silviu Cucerzan, Sujay Kumar Jauhar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-06374
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating Style-Personalized Text Generation: Challenges and Directions Anubhav Jangra, Bahareh Sarrafzadeh, Adrian de Wynter, Silviu Cucerzan, Sujay Kumar Jauhar Abstract While prior research has built tools and benchmarks towards style personalized text generation, there has been limited exploration of evaluation in low-resource author style personalized text generation space. Through this work, we question the effectiveness of the widely adopted evaluation metrics like BLEU and ROUGE, and explore other evaluation paradigms such as style embeddings and LLM-as-judge to holistically evaluate the style personalized text generation task. We evaluate these metrics and their ensembles using our style discrimination benchmark, that spans eight writing tasks, and evaluates across three settings, domain discrimination , authorship attribution , and LLM personalized vs non-personalized discrimination . We provide conclusive evidence to adopt ensemble of diverse evaluation metrics to effectively evaluate style personalized text generation. * * footnotetext: Corresponding authors: [email protected] and [email protected] . $\dagger$ $\dagger$ footnotetext: Work done during an internship at Microsoft. Table 1: Statistics for our extended style personalization evaluation benchmark across the three evaluation settings - domain discrimination (DD) , authorship attribution (AA) , and LLM personalized vs non-personalized (LLM) , with corresponding columns describing the average number of words in the T_{ref} , T_{+} , and T_{-} respectively. Dataset Name #authors Domain Discrimination Authorship Attribution LLM Amazon Food Reviews McAuley and Leskovec 2013 100 157/267/388 161/267/263 267/272/300 ArXiV arXiv.org submitters 2024 100 144/224/447 144/224/229 224/406/505 Blogs Schler et al. 2006 95 173/329/373 164/329/349 329/369/396 Enron Klimt and Yang 2004 100 123/328/520 119/328/334 328/193/219 Lyrics Edenbd 2020 95 210/278/495 211/278/288 278/308/315 Reddit Patel et al. 2022 56 87/38/410 93/38/38 38/222/261 Reuters Lewis 1987 49 363/506/461 366/506/507 506/416/456 Short Stories Carney and Robertson 2019 41 903/1289/267 911/1289/1364 1289/543/515 1 Introduction Recent empirical studies have highlighted a sharp increase in adoption of large language models (LLMs) for writing assistance across various facets of society. LLM-assisted writing has permeated various professional areas like journalism Diakopoulos 2019 , legal services Magesh et al. 2024 , healthcare Baker et al. 2024 ; Bongurala et al. 2024 ; Rengers et al. 2024 , marketing Kumar et al. 2024b , academic writing Khalifa and Albadawy 2024 ; Nguyen et al. 2024 etc . Adoption of AI-based writing assistants have also gained popularity at personal scale, with use cases varying from emails Li et al. 2025 , resume and cover letters Zinjad et al. 2024 , social media posts Long et al. 2023 ; Jain et al. 2023 , blogs Kaisen et al. 2024 etc . This growing reliance on LLMs brings forth the need for style personalized text generation Mysore et al. 2025 . Lack of appropriate techniques to generate text that reflects a user’s writing style can do more harm than good, affecting their interpersonal relationships and professional branding Kadoma et al. 2025 . Recent years have seen increased efforts to utilize the style transferring capabilities of large language models to develop style personalized text generation frameworks, like a personalized prompt rewriter to generate desired style output Li et al. 2024a , linguistic feature controlled multi-attribute style personalized text generation system Alhafni et al. 2024 , and generation calibrated retriever for personalized writing assistance Mysore et al. 2023 . Prior work has also developed generation benchmarks like LaMP Salemi et al. 2023 and LongLaMP Kumar et al. 2024a to to evaluate the style personalization capabilities of text generation models. Yet evaluating the style personalization capabilities of LLMs remains a primarily underexplored problem, with prior efforts majorly adopting rudimentary ngram-overlap based evaluation metrics like BLEU Papineni et al. 2002 , ROUGE Lin 2004 and METEOR Banerjee and Lavie 2005 against a reference text to measure the quality of style personalization in the generated output. In this work, we explore ngram evaluation metrics along with other evaluation paradigms like style-aware embeddings Wegmann et al. 2022 ; Patel et al. 2024 and LLM-as-judge Gu et al. 2024 ; Li et al. 2024b to develop guidelines on evaluation of low-resource style personalized text generation. The major contributions of our work are as follows - 1. We develop an evaluation benchmark spanning eight diverse writing tasks across three different evaluation settings to rigorously evaluate the style discrimination task. 2. To the best of our knowledge, we are the first work to extensively evaluate low-resource style personalized text generation using ngram, style embeddings and LLM-as-judge metrics and their ensembles. 2 Problem Statement Given a reference text T_{ref} , and two candidate texts T_{+} and T_{-} , we define the style discrimination task, where the objective is to identify which of the two candidates is stylistically closer to the reference. Specifically, assuming the existence of an underlying style distribution ( T_{x}\sim S_{x} ), such that S_{ref}\approx S_{+} and S_{ref}\not\approx S_{-} . The task is to learn a discriminator f:(T_{ref},T_{+},T_{-})\mapsto\{+,-\} that predicts which of \{T_{+},T_{-}\} is stylistically closer to T_{ref} . 3 Experimental Settings 3.1 Dataset We extensively evaluate the metrics over eight writing domains across three evaluation settings. We provide the overall statistics of the benchmark and source datasets in Table 1 . We explain the three evaluation settings below, and discuss the dataset construction details in Appendix A . Based on how we obtained the candidates \{T_{+}.T_{-}\} , we propose three evaluation settings. For a reference text T_{ref}\in D^{a_{1}}_{d_{1}} , where D^{a}_{d} denotes set of text written by author a in domain d , we obtain T_{+} and T_{-} as follows - Domain discrimination ( DD ): sample T_{+}\in D^{a_{1}}_{d_{1}} such that T_{+}\neq T_{ref} , and T_{-}\in D^{a_{2}}_{d_{2}} . Authorship attribution ( AA ): sample T_{+}\in D^{a_{1}}_{d_{1}} such that T_{+}\neq T_{ref} , and T_{-}\in D^{a_{2}}_{d_{1}} . LLM personalized vs non-personalized ( LLM ): given a query reconstruction function f_{query}:\mathbb{T}\rightarrow\mathbb{Q} , that outputs a query ( q\in\mathbb{Q} ) corresponding to an input text ( T\in\mathbb{T} ), we obtain the original user query q_{ref}=f_{query}(T_{ref}) . Using a generation model \mathbf{M} , we obtain the candidates T_{+}=\mathbf{M}(q_{ref}|T^{\prime}_{ref}) and T_{-}=\mathbf{M}(q_{ref}) , where T^{\prime}_{ref}\in D^{a_{1}}_{d_{1}} such that T^{\prime}_{ref}\neq T_{ref} (respective prompts in Figures 9 and 4 ). 3.2 Evaluation Metrics We explore three different paradigms of evaluation metrics in this study - Ngram overlap-based evaluation metrics. We explore three widely adopted ngram evaluation metrics, namely BLEU Papineni et al. 2002 , ROUGE Lin 2004 , and METEOR Banerjee and Lavie 2005 . Since these evaluation metrics work on a text pair, we obtain the binary label for similarity measure S as follows: label=\begin{cases}1&\text{if}\hskip 14.22636ptS(T_{ref},T_{+})>S(T_{ref},T_{-})\\ 0&otherwise\end{cases} (1) Style Embedding-based evaluation metrics. We explore two different style embedding models, Wegmann 1 1 1 For simplicity, we refer to this baseline as ”Wegmann” after the first author in Wegmann et al. 2022 . Wegmann et al. 2022 , and StyleDistance Patel et al. 2024 . Since these embedding models map a piece of text into the latent vector space, we use cosine similarity as the similarity measure S to measure similarity between T_{ref} and T_{X},X\in\{+,-\} , and compute the label using Eq. 1 . LLM-as-judge evaluation metrics. We use Ministral-3B , Llama-3.1-8B Grattafiori et al. 2024 , Mistral-24B , Qwen3-32B Yang et al. 2025 , DeepSeek-V3 2 2 2 We use Mistral-24B and DeepSeek-V3 to refer to Mistral-Small-24B-Instruct-2501 and DeepSeek-V3-0324 models respectively. Liu et al. 2024 as our open-source models, and we also