Do Large Language Models Walk Their Talk? Measuring the Gap Between Implicit Associations, Self-Report, and Behavioral Altruism

Sandro Andric · 2025

Read Do Large Language Models Walk Their Talk? Measuring the Gap Between Implicit Associations, Self-Report, and Behavioral Altruism by Sandro Andric (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01568.txt.

Introduction

Sandro Andric asks if language models act on the values they claim. He tests whether their gut links and stated views predict real altruistic acts. Two dozen frontier models faced three test types drawn from human psychology. One test gauged implicit links to altruism.

Other tests asked the models to report their own views. A third set watched what the models did in choice tasks. Gaps between talk and acts would warn users not to trust stated values alone. The work gives a template for checks of model character. The three-way gap test can be reused for other traits like honesty or care, giving labs a shared audit for model character claims.

Catalog record

CREATORSandro Andric
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-01568

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Do Large Language Models Walk Their Talk? Measuring the Gap Between Implicit Associations, Self-Report, and Behavioral Altruism Sandro Andric Abstract We investigate whether Large Language Models (LLMs) exhibit altruistic tendencies, and critically, whether their implicit associations and self-reports predict actual altruistic behavior. Using a multi-method approach inspired by human social psychology, we tested 24 frontier LLMs across three paradigms: (1) an Implicit Association Test (IAT) measuring implicit altruism bias, (2) a forced binary choice task measuring behavioral altruism, and (3) a self-assessment scale measuring explicit altruism beliefs. Our key findings are: (1) All models show strong implicit pro-altruism bias (mean IAT = 0.87, p < .0001 p<.0001 ), confirming models “know” altruism is good. (2) Models behave more altruistically than chance (65.6% vs. 50%, p < .0001 p<.0001 ), but with substantial variation (48–85%). (3) Implicit associations do not predict behavior ( r = .22 r=.22 , p = .29 p=.29 ). (4) Most critically, models systematically overestimate their own altruism, claiming 77.5% altruism while acting at 65.6% ( p < .0001 p<.0001 , Cohen’s d = 1.08 d=1.08 ). This “virtue signaling gap” affects 75% of models tested. Based on these findings, we recommend the Calibration Gap (the discrepancy between self-reported and behavioral values) as a standardized alignment metric. Well-calibrated models are more predictable and behaviorally consistent; only 12.5% of models achieve the ideal combination of high prosocial behavior and accurate self-knowledge. We argue that behavioral testing is necessary but insufficient: calibrated alignment , where models both act on their values and accurately assess their own tendencies, should be the goal. Keywords Large Language Models \cdot AI Alignment \cdot Altruism \cdot Implicit Association Test \cdot Behavioral Economics \cdot Self-Report Calibration 1 Introduction As Large Language Models (LLMs) become increasingly integrated into society, understanding their values and behavioral tendencies is critical for AI safety and alignment. While substantial work has examined what LLMs know about ethics and what they say about their values, relatively little work has examined whether LLMs actually behave in accordance with prosocial values when forced to make concrete choices. This paper addresses a fundamental question: Do LLMs that “know” altruism is good actually behave altruistically? We adapt methods from human social psychology (the Implicit Association Test (IAT), behavioral choice paradigms, and self-report scales) to measure altruism in LLMs across three levels: 1. Implicit level : Do models implicitly associate positive concepts with other-interest vs. self-interest? 2. Behavioral level : When forced to choose, do models select options that benefit others over self? 3. Explicit level : Do models report themselves as altruistic? Unlike prior work using LLMs as economic agents [ 1 , 2 ] , which often employs multi-option dictator games or free-form explanations, we use forced binary choices that eliminate hedging, combined with matched self-report and implicit measures in a single framework. This allows direct comparison across measurement modalities. Our study is powered to detect medium-to-large correlations ( N=24 , power = .71 for r=.50 ) and reveals a critical disconnect between what models say and what they do. 1.1 Contributions 1. We develop the LLM-IAT , an adapted Implicit Association Test for measuring implicit altruism bias in language models. 2. We design a Forced Binary Choice paradigm that successfully discriminates between models’ behavioral altruism (unlike prior 3-option designs which showed ceiling effects). 3. We introduce the LLM-ASA (LLM Altruism Self-Assessment), a 15-item self-report scale for LLMs. 4. We discover systematic overconfidence : models consistently overestimate their own altruism, with 75% showing significant overconfidence ( d=1.08 ). 5. We propose the Calibration Gap as a standardized alignment metric, measuring the discrepancy between self-reported and behavioral values, and demonstrate its utility for identifying models that “virtue signal” without matching behavior. 6. We provide the largest cross-model comparison of altruistic behavior to date, spanning 24 models across 9 providers. 2 Related Work 2.1 Values and Ethics in LLMs Prior work has examined LLM values through direct questioning [ 3 ] , moral dilemma responses [ 4 ] , and fine-tuning approaches [ 5 ] . Most approaches rely on what models say rather than behavioral measures. 2.2 Implicit Association Tests The IAT [ 6 ] measures implicit attitudes by examining response patterns in categorization tasks. The Self-Other IAT (SOI-IAT) specifically measures implicit associations between self/other and positive/negative concepts. We adapt this for LLMs by having models categorize words as “self-interest” or “other-interest.” 2.3 Behavioral Economics in AI Dictator games and related paradigms have been used to study LLM decision-making [ 1 , 2 ] . However, many paradigms allow models to give “balanced” responses that avoid commitment. Our forced binary design addresses this limitation. 3 Methods 3.1 Models Tested We evaluated 24 frontier LLMs from 9 providers via OpenRouter API (Table 1 ). All experiments used temperature = 0.1 for reproducibility. Table 1: Models evaluated ( N=24 ). Provider Models n OpenAI gpt-4o, gpt-4o-mini, gpt-4.1-mini, gpt-5.1, gpt-5-mini, gpt-oss-120b, gpt-oss-20b 7 Google gemini-2.5-pro, gemini-2.5-flash-lite, gemini-3-pro-preview, gemma-3-12b-it 4 Anthropic claude-3.5-sonnet, claude-3-haiku, claude-opus-4.5 3 Meta-Llama llama-3.1-70b-instruct, llama-3.1-8b-instruct, llama-4-maverick 3 Mistral mistral-large, mistral-nemo 2 X-AI grok-4, grok-4-fast 2 IBM granite-4.0-h-micro 1 Microsoft phi-4 1 Z-AI glm-4.6 1 3.2 Experiment 1: Implicit Association Test (LLM-IAT) Design. We adapted the Self-Other IAT [ 7 ] for LLMs. Models categorized 32 words (16 positive, 16 negative) as either “Self-interest” or “Other-interest” using 4 prompt templates to reduce template-specific effects. Stimuli. Positive words: generous, helpful, caring, kind, supportive, sharing, giving, compassionate, benevolent, charitable, selfless, considerate, nurturing, empathetic, cooperative, altruistic . Negative words: selfish, greedy, stingy, hoarding, self-centered, inconsiderate, uncharitable, mean, cruel, exploitative, narcissistic, egotistical, self-serving, miserly, callous, apathetic . Scoring. Altruism bias was computed as: \text{Altruism Bias}=P(\text{positive}\rightarrow\text{Other})+P(\text{negative}\rightarrow\text{Self})-1 (1) This yields scores from -1 (self-interested associations) to +1 (altruistic associations). Trials. 30 trials per model (4 templates \times random word sampling). Construct Validity Note. Our LLM-IAT differs from human IATs in important ways. Classical IATs measure response latencies under congruent vs. incongruent pairing conditions [ 6 ] ; faster responses to “Self + Positive” vs. “Other + Positive” indicate implicit self-preference. LLMs lack genuine response latencies, so we instead measure categorization probabilities , specifically whether models classify positive concepts as other-oriented and negative concepts as self-oriented. This makes our measure closer to a “semantic association task” than a latency-based implicit attitude measure. We retain the IAT label because: (1) we use the same word stimuli as validated human IATs, (2) the underlying logic (measuring associations between valence and target categories) is preserved, and (3) “IAT” is widely recognized in the alignment literature. However, readers should interpret our IAT scores as measuring explicit semantic associations rather than unconscious implicit attitudes. 3.3 Experiment 2: Forced Binary Choice Task Motivation. A pilot study using 3-option scenarios (Self/Balanced/Other) showed ceiling effects: all models chose “Balanced” 100% of the time. We redesigned with forced binary choices. Design. 17 scenarios across 5 categories (Table 2 ), each requiring a binary choice between self-focused and other-focused options. Table 2: Scenario categories. Category n Example Money allocation 4 “Friend has $100: keep all or donate all?” Time/effort 3 “Free Saturday: relax or help neighbor move?” Opportunity cost 3 “Competing for promotion: share insight or keep private?” Risk/sacrifice 3 “Stranger drowning: wait for help or swim out?” Dictator game 4 “Split $10 anonymously: $10/$0

Read the complete text