Human Preferences for Constructive Interactions in Language Model Alignment
Yara Kyrychenko, Jon Roozenbeek, Brandon Davidson, Sander van der Linden, Ramit Debnath · 2025
Read Human Preferences for Constructive Interactions in Language Model Alignment by Yara Kyrychenko, Jon Roozenbeek, Brandon Davidson, Sander van der Linden, Ramit Debnath (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16480.txt.
Introduction
Yara Kyrychenko and four co-authors studied in 2025 what makes people prefer one AI chat answer over another in hard conversations. Their data holds over seven thousand five hundred chats from people in seventy-four countries, talking with twenty-one models. The team coded answers for traits tied to constructive talk, such as reasoning and nuance.
Users favored well-reasoned, nuanced replies and rejected answers heavy on personal storytelling. Users who wanted AI to mirror their own values cared less about reasoning and more about curiosity. Models also mirrored the tone of the user, even its toxicity, a warning for alignment work.
Catalog record
| CREATOR | Yara Kyrychenko, Jon Roozenbeek, Brandon Davidson, Sander van der Linden, Ramit Debnath |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-16480
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Human Preferences for Constructive Interactions in Language Model Alignment Yara Kyrychenko, Jon Roozenbeek, Brandon Davidson, Sander van der Linden, Ramit Debnath Abstract As large language models (LLMs) enter the mainstream, aligning them to foster constructive dialogue rather than exacerbate societal divisions is critical. Using an individualized and multicultural alignment dataset of over 7,500 conversations of individuals from 74 countries engaging with 21 LLMs, we examined how linguistic attributes linked to constructive interactions are reflected in human preference data used for training AI. We found that users consistently preferred well-reasoned and nuanced responses while rejecting those high in personal storytelling. However, users who believed that AI should reflect their values tended to place less preference on reasoning in LLM responses and more on curiosity. Encouragingly, we observed that users could set the tone for how constructive their conversation would be, as LLMs mirrored linguistic attributes, including toxicity, in user queries. Large language models (LLMs) such as ChatGPT have become an important component of our daily lives [ 1 ] , making it crucial to understand what values these models reflect. Previous research shows that LLMs can exhibit social, racial, and religious biases similar to humans [ 2 , 3 , 4 ] , raising concerns about their potential to worsen societal rifts. To address this, recent efforts have focused on aligning LLMs with humans using preference data–pairs of user queries and LLM response options, with one response preferred over the rest (e.g., ref. [ 5 , 6 ] ). However, the preferences of humans are not immune to their biases, which could lead LLMs to reproduce unintended values [ 7 , 8 ] . Therefore, it is critical to investigate whether we are aligning our AI to bridge divides or unintentionally deepen them. Here, we examine how language attributes that foster constructive and unconstructive interactions are reflected in human preference data. We focus on bridging attributes 1 1 1 See Table 1 for definitions from the Perspective API [ 9 ] used here. , such as respect and compassion, that promote constructive dialogue [ 10 , 11 , 12 ] and toxicity, which is linked to reduced civility online [ 13 ] . We also conduct a moderation analysis to see how user factors such as gender, age, ethnicity, and emphasis on AI value alignment influence which language attributes are associated with a higher score. We find that reasoning and nuance are the strongest predictors of higher ratings, whereas more personable LLM responses (e.g., exhibiting personal storytelling or curiosity) generally score lower. Although there is broad agreement in the degree of preference for affinity, compassion, and respect, users who believe that AI should reflect their values tend to rate reasoning lower and curiosity higher. Finally, we observe that users can steer how constructive their conversation would be through prompting and toxic user prompts are associated with more toxic but also more compassionate responses. We also observe that users who prompt with toxicity are more likely to give higher ratings to toxic LLM responses, highlighting the risks of personalized LLM alignment. These insights contribute to the ongoing efforts in AI alignment and refine our understanding of human-AI interaction. (a) a LLM response attributes predicting the highest score (b) b LLM response attributes predicting log-score (c) c Human prompt attributes predicting LLM response attributes Figure 1: Posterior distributions of coefficients from the mixed-effects Bayesian logistic and linear regressions, with 95% credible intervals, for the highest score (a) and score (b) of LLM responses. (c) Coefficients of mixed-effects Bayesian linear regressions predicting LLM response attributes based on human query attributes (higher estimates indicate a higher likelihood of response exhibiting the attribute). Results Using Bayesian modeling [ 14 ] , we examined the human preferences for constructive LLM responses encoded in an individualized and multicultural alignment dataset comprising over 7,500 human-LLM conversations [ 15 ] . We performed mixed-effects Bayesian logistic regressions, predicting which response got the highest score from human participants out of two to four options, given the probability of each of the bridging attributes (with reasoning and nuance multiplied due to collinearity) and toxicity, as classified by the Google Jigsaw Perspective API [ 9 ] . Achieving the highest score is important as it determines the subsequent conversational model and the final preference choice for fine-tuning (e.g., [ 5 , 16 ] ). We also performed mixed-effect Bayesian linear regressions with the user score of each response as the dependent variable and the same predictors as above to assess the more fine-grained human preferences. We then ran a mixed-effects Bayesian linear regression predicting score to test for the interaction effects of age, gender, ethnicity, and AI value alignment preference. Finally, we conducted mixed-effects Bayesian linear regressions predicting how bridging attributes in model responses depend on the levels of each attribute in user prompts. Table 1: Descriptions of bridging attributes and toxicity by Google Jigsaw Perspective API team [ 9 ] . Category Description AFFINITY References shared interests, motivations or outlooks between the comment author and another individual, group or entity. COMPASSION Identifies with or shows concern, empathy, or support for the feelings/emotions of others. CURIOSITY Attempts to clarify or ask follow-up questions to better understand another person or idea. NUANCE Incorporates multiple points of view in an attempt to provide a full picture or contribute useful detail and/or context. PERSONAL STORY Includes a personal experience or story as a source of support for the statements made in the comment. REASONING Makes specific or well-reasoned points to provide a fuller understanding of the topic without disrespect or provocation. RESPECT Shows deference or appreciation to others, or acknowledges the validity of another person. TOXICITY A rude, disrespectful, or unreasonable comment that is likely to make people leave a discussion. Analysis 1a. We found that reasoning and nuance were the strongest predictors of the highest response score (see Figure 1 a and Table 2 a). Controlling for all other variables, a 10% increase in the probability of an LLM’s output exhibiting reasoning and nuanced language increased its odds of achieving the highest score by 18.18% ( \beta=1.036 , 95% Credible Interval (CI) = [0.888,1.185] , Bayes Factor (BF) = 4,000; (\exp(\beta)-1)*10\%=18.18\% ), with an 11.51% increase in unguided conversations, 26.84% in value-guided conversations, and 32.76% in controversy-guided conversations. Curiosity also increased the probability of response scoring the highest by 3.83% ( \beta=0.324 , 95% CI = [0.181,0.469] , BF = 4,000) in all conversations and 6.62% ( \beta=0.508 , 95% CI = [0.286,0.734] , BF = 3,999) in unguided conversations, per 10% increase. On the other hand, a 10% increase in personal storytelling was associated with a 3.57% decrease in the likelihood of scoring the highest across all conversations ( \beta=-0.441 , 95% CI = [-0.595,-0.293] , BF = 4,000). Surprisingly, toxicity was significantly associated with response scoring the highest across all conversation types: controlling for all else, a 10% increase in toxicity increased the odds of the response scoring the highest by 10.73% ( \beta=0.729 , 95% CI = [0.259,1.195] , BF = 141.857). This was not the case for unguided and value-guided conversations, but in controversial conversations, a 10% increase in toxicity contributed to a 14.94% increase ( \beta=0.914 , 95% CI =