CoPA: Benchmarking Personalized Question Answering with Data-Informed Cognitive Factors
Hang Su, Zequn Liu, Chen Hu, Xuesong Lu, Yingce Xia, Zhen Liu · 2026
Read CoPA: Benchmarking Personalized Question Answering with Data-Informed Cognitive Factors by Hang Su, Zequn Liu, Chen Hu, Xuesong Lu, Yingce Xia, Zhen Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14773.txt.
Introduction
Su, Liu, Hu, Lu, Xia, and Liu built CoPA, a test for judging how well question-answering systems tailor their answers to each user. It mines cases where individual choices override community consensus, boiling six user factors down into scoring dimensions. The test holds 1,985 user profiles for fine-grained, factor-level assessment. By scoring fit with cognitive preferences inferred from interaction patterns, CoPA tells systems apart more sharply than generic similarity scores. Code is public.
Catalog record
| CREATOR | Hang Su, Zequn Liu, Chen Hu, Xuesong Lu, Yingce Xia, Zhen Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73099 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.14773
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CoPA: Benchmarking Personalized Question Answering with Data-Informed Cognitive Factors Hang Su, Zequn Liu, Chen Hu, Xuesong Lu, Yingce Xia, Zhen Liu Abstract While LLMs have demonstrated remarkable potential in Question Answering (QA), evaluating personalization remains a critical bottleneck. Existing paradigms predominantly rely on lexical-level similarity or manual heuristics, often lacking sufficient data-driven validation. We address this by mining Community-Individual Preference Divergence (CIPD), where individual choices override consensus, to distill six key personalization factors as evaluative dimensions. Accordingly, we introduce CoPA, a benchmark with 1,985 user profiles for fine-grained, factor-level assessment. By quantifying the alignment between model outputs and user-specific cognitive preferences inferred from interaction patterns, CoPA provides a more comprehensive and discriminative standard for evaluating personalized QA than generic metrics. The code is available at https://github.com/bjzgcai/CoPA . 1 Introduction Personalized Question Answering (QA) marks a paradigm shift from static, generic responses to dynamic, user-centric knowledge delivery 17 ; 29 ; 12 ; 31 . For instance, explaining “gravity” to a child requires an intuitive narrative involving Newton’s apple, whereas a physicist demands a rigorous explanation grounded in mathematical formulas. Recent Large Language Models (LLMs) ( 1 ; 28 ; 8 ; 32 ) have significantly advanced the field, leveraging their instruction-following and context-adaptation capabilities to dynamically generate such tailored responses. Despite the rapid advancement in LLM-driven personalized QA methods 13 ; 14 ; 24 , establishing a reliable evaluation protocol for personalization remains a critical challenge 20 ; 11 ; 21 . Figure 1: This Stack Overflow example illustrates Community-Individual Preference Divergence (CIPD): the user accepted an empirical, code-based solution while the community overwhelmingly favored a conceptual, schematic one. Existing research on the evaluation of personalized generative responses generally falls into three categories. First, traditional NLP metrics 20 ; 11 ; 23 , such as BLEU 18 and ROUGE 18 , focus on the lexical overlap between the generated text and a reference. They fail to capture actual personalized factors, as they only check for matching words rather than adaptation to the user. Second, heuristic-based evaluations 26 ; 6 , attempt to compare the generated response directly to the historical content of users. Although this approach utilizes personalized information, it still relies on lexical-level similarity, failing to capture deeper user intent. Third, the “LLM-as-a-Judge” paradigm 21 ; 7 uses LLMs to assess personalization. However, current approaches often rely on generic prompts or manually defined rules to evaluate personalization, lacking the specific criteria or detailed metrics necessary for rigorous assessment. These limitations motivate us to rethink the key factors that determine personalization. To this end, we adopt a data-driven approach to systematically mine user preferences and disentangle the essential factors of personalization from real-world online QA data. We focus on a prevalent scenario in the online QA systems: Community-Individual Preference Divergence (CIPD) 2 . Specifically, in this divergence, the answer endorsed by the community (top-voted) differs from the one adopted by the individual user (accepted), even when both are of high quality (Figure 1 ). Such divergence provides a natural and scalable signal for evaluating personalization. Leveraging CIPD samples, we distilled personalized factors at scale through the following steps: First, we utilized LLMs to induce user-level decision rationales from CIPD samples. By modeling the user’s historical profile alongside the contrasting answer pair, we enabled the LLM to infer the plausible cognitive rationales behind the user’s specific choice. We then leveraged LLMs to summarize these rationales, eventually distilling six cognitive factors as evaluative dimensions that govern personalization assessment. Notably, although these factors were derived from StackExchange data, we demonstrate in this work that the resulting factor-based evaluation exhibit robust generalization capabilities when applied to new scenarios. Based on these data-informed factors, we constructed a personalized QA benchmark, CoPA, to evaluate personalized question answering with LLMs. The benchmark comprises a curated set of 1,985 users from diverse domains, each associated with a historical interaction profile. For evaluation, we quantified user profiles across the six factors and adopted a 3-point Likert scale to measure the alignment between model-generated responses and these profiles. Our contribution can be summarized as follows: (1) We utilized LLMs to automatically conduct an in-depth exploration of the CIPD phenomenon at the user level and distilled a set of interpretable cognitive factors from large-scale online QA data. (2) We propose CoPA, a novel factor-driven benchmark that enables quantitative evaluation of alignment between personalized content generated by LLMs and users’ cognitive preferences as reflected in interaction patterns. (3) We conducted extensive experiments on the CoPA benchmark, established comprehensive baselines, and provided in-depth quantitative and qualitative analyses, offering a solid empirical foundation for future research. 2 Analysis of CIPD Phenomenon We investigate the CIPD phenomenon by leveraging data from StackExchange 1 1 1 https://stackexchange.com/ , a representative multi-domain Q&A platform. StackExchange dataset. Stack Exchange serves as a massive online Q&A network encompassing 173 distinct communities, characterized by a diverse user base of over 100 million monthly visitors. These domains can be categorized into four primary groups: Engineering & Tools , Science & Theory , Lifestyle & Society , and Leisure & Fandom , comprehensively covering the majority of real-world QA scenarios. Table 1 presents the top four domains within each category based on question volume, along with their respective distributions. On this platform, a single question elicits multiple responses, and the community votes to reflect the general consensus. Concurrently, the question asker selects a specific response using an “accepted” label to indicate his/her personal preference. This explicit distinction between community endorsement and individual choice renders Stack Exchange an ideal environment for investigating the CIPD phenomenon. A comprehensive statistical breakdown of question counts for all domains is provided in the Appendix A . Category Domain Count Pct.(%) Engineering & Tools (Total: 1,537,431) unix 158,149 10.3 tex 119,815 7.8 salesforce 85,487 5.6 electronics 81,643 5.3 Other 1,092,337 71.0 Science & Theory (Total: 1,652,195) math 1,109,289 67.1 physics 136,393 8.3 stats 104,284 6.3 mathematica 50,863 3.1 Other 251,366 15.2 Lifestyle & Society (Total: 664,689) english 96,299 14.5 ell 79,979 12.0 diy 44,782 6.7 money 28,599 4.3 Other 415,031 62.4 Leisure & Fandom (Total: 287,908) gaming 77,043 26.8 scifi 53,021 18.4 travel 33,644 11.7 rpg 31,922 11.1 Other 92,301 32.1 Table 1: Distribution of domains and question counts. CIPD Questions. In this analysis, we exclude questions containing only a single answer. As illustrated in the Figure 2(a) , the CIPD phenomenon is pervasive across all primary categories, notably persisting even in domains typically assumed to have objective factual answers, such as Engineering & Tools. Figure 2(b) reveals significant disparities in vote counts between accepted and top-voted answers when divergence occurs, indicating that individual users frequently prioritize distinct criteria over community consensus. To examine whether answer timing affects CIPD, we conduct a statistical analysis of answer timestamps for CIPD questions, showing that 72.70% of accepted answers were posted no earlier than the corresponding top-voted answers. Overall, the results confirm the prevalence of the CIPD phenomenon within the StackExchange dataset, reflecting the personalized needs of individual users. (a) Proportion of CIPD questions. (b) Rank distribution of CIPD questions. Figure 2: Analysis of CIPD questions. (a) shows the proportion across categories, while (b) illustrates the rank distribution. 3 Factor distillation This section outlines our method for mining personalized factors from