Modeling Community Attitude through Reaction Tone: A Human-AI Collaborative Framework for Evaluating LLM Alignment with Linguistic Behaviors in Online Communities
Nuan Wen & Xuezhe Ma · 2026
Read Modeling Community Attitude through Reaction Tone: A Human-AI Collaborative Framework for Evaluating LLM Alignment with Linguistic Behaviors in Online Communities by Nuan Wen & Xuezhe Ma (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27388.txt.
Introduction
Nuan Wen, Chanbin Lim, and Xuezhe Ma test whether language models can mimic how online communities react to news. Their CARE framework benchmarks simulated discourse against real reactions from 207 Reddit communities. Community context helps at the broad level but collapses on specific events, and gains in some communities come with losses in others.
Catalog record
| CREATOR | Nuan Wen & Xuezhe Ma |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37977 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.27388
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Modeling Community Attitude through Reaction Tone: A Human-AI Collaborative Framework for Evaluating LLM Alignment with Linguistic Behaviors in Online Communities Nuan Wen & Xuezhe Ma Abstract Large language models (LLMs) are increasingly utilized as proxies for computational social analysis; yet, their ability to faithfully represent the ”thick descriptions” ( Geertz 1973 ) of human communities remains a critical challenge. Current evaluations often reduce social identity to static labels, sidelining how real-world groups navigate social shifts. To bridge this gap, we introduce CARE ( C ommunity- A ware R eaction E valuation), a reaction-centered framework that benchmarks LLM-simulated discourse against the authentic, event-contingent responses of distinct communities to real-world news. By characterizing a fine-grained spectrum of illocutionary tones and the underlying attitudes they manifest—validated through human-AI collaboration—our diagnosis reveals a persistent ”realism gap”: steering LLMs with explicit community prompts fails to inherently improve simulation fidelity. Analysis further identifies divergent behavioral signatures among frontier models, suggesting that current alignment strategies remain insufficient for capturing the sociolinguistic dynamics of online groups. 1 Introduction Large language models (LLMs) are increasingly deployed as generative proxies for social analysis, celebrated for their ability to encode the linguistic contours of social norms and scale the simulation of human dynamics ( Anthis et al. 2025 ; Piao et al. 2025 ) . Yet, as their application in computational social science expands, a critical disconnect has emerged between how models are aligned to represent social groups and how these groups authentically function ( Hovy & Yang 2021 ; Kabir et al. 2025 ; Shen et al. 2025 ; Wang et al. 2025 ) . Anthropological and sociolinguistic traditions emphasize that communities are not merely structural graphs or demographic buckets, but dynamic discursive spaces characterized by ”thick” realities ( Geertz 1973 ; Adilazuarda et al. 2024 ) . Within these spaces, group identity is not a static property but an active, event-contingent performance, negotiated through linguistically mediated reactions a community adopts to articulate its collective stance. However, recent socio-cultural alignment frameworks for LLMs remain decoupled from this dynamic reality. Current evaluations typically rely on coarse abstractions, mapping complex social identities to predefined demographic categories and benchmarking them against aggregated survey data ( Santurkar et al. 2023 ; Tao et al. 2024 , e.g.) . By treating language as a static semantic repository, these approaches capture what a community stands for but risk overlooking how such stances are pragmatically expressed in real-world discourse. For example, while a community can be semantically categorized as ”supportive” of some social value, the manner of that support—whether expressed through weary resignation, sarcastic compliance, or communal solidarity—depicts its social reality. We argue that a community’s unique stance is fundamentally encoded in its reaction tones, as these linguistic signals provide a critical dimension for evaluating socio-cultural alignment. To account for these attitudinal nuances, we shift the evaluation of community-aware socio-cultural alignment from static identity labels to a reaction-centered paradigm and introduce a new benchmark and framework: C ommunity- A ware R eaction E valuation ( CARE ). Anchored in the COVID-19 pandemic—a defining period of global fragility encapsulating a dense spectrum of shared uncertainty and localized struggles ( Bavel et al. 2020 ; Hosseinzadeh et al. 2022 ) — CARE measures whether LLM-simulated discourse aligns with authentic, event-contingent linguistic behaviors observed in online communities. To operationalize this framework, we pair real-world news articles from this era with 3,749 reactions across 207 diverse Reddit communities, spanning four continents and ten thematic domains. Consequently, CARE provides a rigorous environment for benchmarking how frontier models navigate the ”thick” reality of collective human experience. As illustrated in Figure 1 , our evaluation study focuses on the diagnostic delta \Delta : the distributional shift in a model’s generated tone and attitude when transitioning from a ”community-blind” baseline to a ”community-informed” simulation. We hypothesize that tracing this shift reveals whether injected community information successfully nudges a model toward a realistic evaluative pulse. Our multi-level strategy examines both instance-level fidelity and distributional resemblance across communities. This design enables a diagnostic interpretation of how model behavior shifts, redistributes, or concentrates, avoiding a reductionist view of alignment success based on isolated metrics. Figure 1: Conceptual view of the CARE framework. Ground-truth community reactions are compared against both community-blind and community-informed LLM simulations. 2 Related Work While the literature on large language model (LLM) alignment is vast, we constrain our focus to the two domains most critical to our framework: the evaluation of socio-cultural alignment and the application of lexicon-based analysis to social meaning. Broad overviews of general alignment paradigms and safety mechanisms are extensively covered in recent surveys ( Shen et al. 2023 ; Sucholutsky et al. 2023 ; Lu et al. 2025 ) . To evaluate whose values models actually reflect, prompt-based steering has emerged as a highly effective foundational tool for measurement, allowing researchers to probe model biases by conditioning generations on specific demographic or ideological personas ( Santurkar et al. 2023 ; He et al. 2024 ; Wu et al. 2025 ) . Building on these measurement paradigms, the field has increasingly shifted toward assessing socio-cultural alignment ( Adilazuarda et al. 2024 ; Chiu et al. 2025 ; Liu et al. 2025a ) . Many of these foundational approaches quantify alignment by benchmarking model generations against predefined cultural dimensions and established sociological surveys ( Santurkar et al. 2023 ; Tao et al. 2024 ; Masoud et al. 2025 ; Rao et al. 2025 ; Sukiennik et al. 2025 ) . However, recognizing that static labels and closed-style evaluations often fail to capture the dynamic reality of these social groups ( Hovy & Yang 2021 ; Röttger et al. 2024 ; Kabir et al. 2025 ; Shen et al. 2025 ) , subsequent work such as Liu et al. 2025b explores dynamic cultural adaptation through role-play-inspired simulation and cultural learning pipelines. Closest to our objective is the emerging subfield of community-level alignment ( Shi et al. 2024 ; Lin & Wei 2026 ; Chu et al. 2025 ) . While these approaches capture broad community archetypes ( Prinster et al. 2024 ) , modeling the authentic, event-contingent reactions that characterize how real-world online groups navigate complex social shifts remains an open frontier. To bridge this realism gap, we draw on the long-standing tradition of lexicon-based analysis in NLP, which offers a robust, interpretable mechanism for evaluating text beyond surface-level generation. Extensive prior work has utilized linguistic schemata to tackle a wide array of foundational NLP and computational linguistic tasks ( Stolcke et al. 2000 ; Pennebaker et al. 2007 ; Ding et al. 2008 , e.g.) , while more recently, lexicon-driven approaches have evolved to tackle complex tasks that probe deeper into social meaning ( Havaldar et al. 2024 ; Huang et al. 2024 ) and inspired this work. 3 Corpus and Dataset Construction The CARE corpus comprises all 2020 Reddit posts containing either of the query words ”coronavirus” or ”covid”. We restrict content to English to isolate the dataset from multilingual effects. Table 1 reports dataset statistics for each processing stage detailed below. Processing Stage # subreddits # articles # keyphrases # posts # reactions avg. react. len. (Initial Corpus) 4,206 117,747 199,399 141,487 652,429 40.28 Subreddit Selection 207 99,531 145,274 117,478 547,090