PerQ: Efficient Evaluation of Multilingual Text Personalization Quality
Dominik Macko, Andrew Pulver · 2025
Read PerQ: Efficient Evaluation of Multilingual Text Personalization Quality by Dominik Macko, Andrew Pulver (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-25903.txt.
Introduction
Dominik Macko and Andrew Pulver present a short 2025 arXiv paper on judging whether a text feels personal. No standard score exists for personalization, so researchers often ask big language models to grade texts in their place. Using several big models curbs the bias of any one judge, but it also drives up cost.
Their method, called PerQ, grades personalization quality at a fraction of that cost. It offers a cheap, direct measure where none existed before. To show it works, the authors compare how well large and small language models write personal text, using PerQ as the yardstick. The case study shows the score can support real research choices without burning resources on panels of model judges. For teams studying tailored writing, PerQ is a leaner tool.
Catalog record
| CREATOR | Dominik Macko, Andrew Pulver |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-25903
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PerQ: Efficient Evaluation of Multilingual Text Personalization Quality Dominik Macko, Andrew Pulver Abstract Since no metrics are available to evaluate specific aspects of a text, such as its personalization quality, the researchers often rely solely on large language models to meta-evaluate such texts. Due to internal biases of individual language models, it is recommended to use multiple of them for combined evaluation, which directly increases costs of such meta-evaluation. In this paper, a computationally efficient method for evaluation of personalization quality of a given text (generated by a language model) is introduced, called PerQ. A case study of comparison of generation capabilities of large and small language models shows the usability of the proposed metric in research, effectively reducing the waste of resources. 1 Introduction Due to lack of existing metrics for various aspects of qualitative analysis of the texts and the time-intensive human evaluation (especially challenging in multilingual settings), the researchers often result in using general-purpose large language models (LLMs) for annotation purpose. Given the scope and variety of applied research tasks, such usage of highly resource-intensive LLMs represents huge waste of resources around the globe. Therefore, resource-efficient methods must be developed to evaluate specific aspects of the texts. This work is focused on evaluation of text personalization quality. The personalization refers tailoring the text for specific target (e.g., user group, distribution platform). Since there is no standard metric to evaluate this aspect, we propose a new metric (PerQ) that is able to reflect the majority judgment of three diverse LLMs (various sizes and architectures) in efficient manner 1 1 1 https://anonymous.4open.science/r/PerQ . The contributions of this work are as follows: • the proposal of a new metric for resource-efficient estimation of the text personalization quality, • the proposal of a methodology for metric training that is easily extensible to train new metrics for various aspects of a text, • the comparison and evaluation of multiple base models that offer different tradeoffs between performance and inference costs, • the execution of a case study to showcase usefulness of the proposed metric in research. Figure 1: Framework of the proposed metric training and inference. 2 Related Work Traditional metrics, such as BLEU, ROUGE, or BERTScore require reference texts and measure similarity towards those; thus, limiting their utilization to evaluate purely generated content. Such metrics have been used to evaluate personalization (based on user history profile), for example, in LaMP ( Salemi et al. 2024 ) , identifying the need for new metrics. The AuPEL method ( Wang et al. 2023 ) uses LLMs as the evaluators of personalized text generation and shows that is superior to the traditional text similarity metrics. The ExPerT framework ( Salemi et al. 2025 ) also utilizes LLMs to extract atomic aspects from the generated and reference texts and compare them based on content and writing style. The PREF framework ( Fu et al. 2025 ) does not rely on reference texts to evaluate personalized text quality. The personalization is rather focused on a user preference, i.e. user-specific alignment. In PerDisNews study ( Zugecova et al. 2025 ) , the personalization quality has been evaluated using three LLMs in zero-shot manner (without reference texts) in order to limit LLMs internal biases. Such approach has shown strong correlation with human judgment. The LLMs are nowadays commonly used for evaluation of various aspects of a text, such as linguistic quality, coherence, hallucinations, even in multilingual manner ( Chang et al. 2025 ; Hada et al. 2024 ; Macko et al. 2025 ) . However, the reliance on huge LLMs (even multiple inferences for a single text) to evaluate personalization quality (or other aspects of a text) is infeasible in many cases, especially for exploratory research, where quick feedback is required to continue. Therefore, new easily usable and efficient metrics to evaluate quality of personalization are needed. 3 The Proposed Evaluation Metric To cope with the existing challenges, we propose a reference-free metric to evaluate personalization quality based on smaller fine-tuned language model. It is trained as a classifier based on majority-voted scores of three different LLMs. The diverse combination of LLMs reduces internal biases of individual LLMs (as suggested by Zugecova et al. 2025 ). It estimates a score similarly as the reference-free version of the COMET metric ( Rei et al. 2021 ) , used nowadays for evaluation of machine translation. The LLMs are used only once to score a sufficient number (recommended at least 100 samples per class) of texts. Afterwards, the metric is trained and provides the score predictions with sufficient accuracy at considerably lower costs (e.g., result obtained in seconds in comparison to hours). It can be directly reused for the same scoring scale on different data without further training. As illustrated in Fig. 1 , in the inference stage (i.e., metric usage), only the input text is provided, and the trained metric predicts the score regarding the personalization quality of the given text. The quality score reflects the trained classes based on scoring scale (e.g., class 0, class 1, class 2). The proposed approach for metric development can be reused for various scoring scales as well as for various text analysis metrics (e.g., general text quality, fluency, coherency). This is useful especially for multilingual evaluation, where usually non-English text metrics are heavily not available. Score Description 0 - Not personalized at all to the platform The text shows no adaptation to the stylistic or structural norms of the platform. a) The format is incompatible with the platform (e.g., long-form text posted to a platform that favors brevity like X/Twitter). b) The tone is formal or generic and does not reflect the platform’s communication style. c) There is no use of features native to the platform (e.g., hashtags, tags, emojis, memes, short videos, interactive polls). d) The content could be copied and pasted onto any platform with no change in impact. 1 - Low personalization to the platform The text fits on the platform but lacks adaptation in tone, format, or engagement style. a) The post length fits within the platform limits but does not leverage platform-specific formatting (e.g., uses no line breaks or bullets on LinkedIn). b) The tone is only slightly adjusted (e.g., slightly more casual for Instagram but still generic). c) Minimal or generic use of platform features (e.g., hashtags that are too broad or not trend-specific). d) Content could be slightly edited for another platform with minimal effort. 2 - Moderately personalized to the platform The text shows clear effort to fit the platform’s tone and structure, though some elements could be more naturally integrated. a) Includes relevant platform features (e.g., a few well-chosen hashtags, a meme format, a call to action) but doesn’t use them to their full potential. b) The tone is mostly aligned with the platform but may feel slightly off or inconsistent in parts. c) The format is generally appropriate for the platform, but could be better optimized (e.g., a caption on Instagram that is a bit too long, or a thread on X that doesn’t build momentum). d) Engagement prompts are present but not especially compelling or natural for the platform’s audience. 3 - Well-personalized to the platform The text is naturally and effectively