Machine individuality: Separating genuine idiosyncrasy from response bias in large language models

Valentin Kriegmair, Dirk U. Wulff · 2026

Read Machine individuality: Separating genuine idiosyncrasy from response bias in large language models by Valentin Kriegmair, Dirk U. Wulff (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16755.txt.

Introduction

Kriegmair and Wulff ask whether behavioral differences among language models reflect stable individuality or global response biases and noise. They apply crossed random-effects models from psychometrics to seventy-four point nine million ratings from 10 open-weight models. The ratings cover over 100,000 words across 14 psycholinguistic norms. On average about 17 percent of variance traces to stimulus-specific individuality, beating a statistical null model clearly. Cross-norm prediction shows individuality as a coherent fingerprint for each model, not attributable to bias or noise.

Catalog record

CREATORValentin Kriegmair, Dirk U. Wulff
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38753
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.16755

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Machine individuality: Separating genuine idiosyncrasy from response bias in large language models Valentin Kriegmair, Dirk U. Wulff Abstract As large language models (LLMs) are increasingly integrated into daily life, in roles ranging from high-stakes decision support to companionship, understanding their behavioral dispositions becomes critical. A growing literature uses psychometric inventories and cognitive paradigms to profile LLM dispositions. However, these approaches cannot determine whether behavioral differences reflect stable, stimulus-specific individuality or global response biases and stochastic noise. Here, we apply crossed random-effects models—widely used in psychometrics to separate systematic effects—to 74.9 million ratings provided by 10 open-weight LLMs for over 100,000 words across 14 psycholinguistic norms. On average, 16.9% of variance is attributable to stimulus-specific individuality, robustly exceeding a statistical null model. Cross-norm prediction analyses reveal this individuality as a coherent fingerprint, unique to each model. These results identify individual differences among LLMs that cannot be attributed to response biases or stochastic noise. We term these differences machine individuality . Keywords: Artificial intelligence ∣ \mid large language models ∣ \mid personality ∣ \mid machine individuality ∣ \mid psycholinguistic norms As large language models (LLMs) are deployed for a widening range of purposes, from high-stakes decision support to everyday companionship, understanding their behavioral dispositions becomes consequential. Whether a model renders moral judgments harshly or gently, or rates emotional content vividly or flatly, shapes its usability and performance. Acknowledging this point, major providers now offer models with distinct personality modes. A growing body of work has begun to address the issue by applying psychometric inventories ( 1 , 2 ) , cognitive paradigms ( 3 , 4 ) , and classification methods ( 5 ) to characterize the behavioral dispositions of LLMs. These efforts share an implicit assumption—that LLM behavior varies in stable, measurable ways across models—for which evidence remains surprisingly weak. Measured “personality” scores shift substantially with prompt template ( 6 ) , option ordering ( 7 ) , and scale type ( 8 ) . Here, we ask a more fundamental question: Does stable behavioral individuality—separable from shared consensus, response biases, and stochastic noise—exist in LLMs at all? This requires separating sources of variation that existing approaches conflate—a challenge personality psychology has long recognized. Behavioral differences can reflect substantive traits (stable, stimulus-specific dispositions) or response styles such as acquiescence and extreme responding that shift all responses directionally ( 9 , 10 ) . Yet individuality resides in person-by-situation interactions. The same ambiguity afflicts current characterizations of LLM behavior. Most compellingly, Sun et al. ( 5 ) show that a fine-tuned classifier identifies source LLMs with 97% accuracy, even after paraphrasing. Yet their own analyses reveal that the discriminative signal is carried predominantly by surface-level stylistic regularities, such as characteristic phrases or formatting conventions, features that function as the machine analogue of response styles. In short, existing approaches can tell models apart but cannot determine whether the differences reflect situation-specific evaluation or general response styles. Figure 1: Machine Individuality: Variance Partitioning and Specificity. A. Crossed random-effects decomposition of \sim 5.4M ratings per norm (10 open-weight LLMs, 5 repetitions) across 14 psycholinguistic norms (five sensory norms and two Age of Acquisition norms grouped for brevity). Variance decomposes into Trait (consensus), Bias (model offset), Idiosyncrasy (Model \times Word), and Residual. Idiosyncrasy averages 16.9% (4.8%–34.0% across norms). B. Ridge regression specificity ratios: Each model’s per-word idiosyncrasy estimates on a held-out norm predicted from the remaining 13, relative to cross-model baselines. All ratios exceed 1.0 on average (1.74–3.43), confirming coherent, model-specific semantic fingerprints. To untangle situation-specific individuality from directional bias, we turn to crossed random-effects models ( 11 ) , the standard psychometric tool for decomposing observed variation into systematic effects, such as shared consensus, systematic biases, and the person-by-situation interaction that constitutes individuality. This decomposition requires a fully crossed design—every respondent responds to every situation. Standard psychometric inventories are ill-suited: They were designed for human self-report, and their construct validity does not transfer to LLMs ( 12 ) . What would constitute a valid behavioral assessment for LLM individuality is far from obvious. Here, we propose psycholinguistic word ratings—numerical judgments along dimensions such as valence, arousal, concreteness, or morality—as a novel approach. These ratings were obtained for over 100,000 words across virtually the entire semantic space. By rating this broad lexicon, a model effectively reveals how it would evaluate virtually any situation. We administered zero-shot rating tasks to 10 open-weight LLMs spanning a broad range of architectural implementations and parameter scales (20B–235B), eliciting ratings across 14 psycholinguistic norms (see Figure 1 A) via repeated stochastic sampling ( \sim 74.9 million total observations). A crossed random-effects model then decomposed each rating y_{ijk} (word i , model j , repetition k ): y_{ijk}=\mu+\tau_{i}+\beta_{j}+\iota_{ij}+\epsilon_{ijk} (1) into shared Trait ( \tau_{i} ), systematic model Bias ( \beta_{j} ), Residual noise ( \epsilon_{ijk} ), and—critically—model Idiosyncrasy ( \iota_{ij} ). This interaction captures how each model evaluates each word, net of consensus and bias. If it accounts for zero variance, apparent behavioral differences reduce to Bias and Residual; conversely, if it is substantial, and larger than Bias, models possess genuine individuality. Results Figure 1 A displays the crossed random-effects decomposition of \sim 5.4M ratings per norm across 14 psycholinguistic norms, with the five sensory norms (Auditory, Gustatory, Haptic, Olfactory, and Visual) grouped. Shared Trait variance dominates (32.1%), confirming broad cross-model consensus on the semantic landscape ( 13 ) . The critical question is how the remaining variance partitions between systematic Bias and stimulus-specific Idiosyncrasy. The answer is unambiguous: Idiosyncrasy accounts for 16.9% of total variance on average, ranging from 4.8% (Valence) to 14.3% (across sensory norms; 9.7%–18.0% individually) to 34.0% (Gender Association), and robustly exceeds parametric null expectations across all 14 norms (null upper bound <0.033\% ; all p<0.01 ). Models differ not only in their general response tendencies, but in how they evaluate specific words. Crucially, these idiosyncrasies are not isolated to individual dimensions but form internally coherent, model-specific fingerprints. Using Ridge regression to predict each model’s best linear unbiased predictions (BLUPs)—i.e., the estimated per-word deviations \hat{\iota}_{ij} after removing shared trait and bias effects—on a held-out norm from its BLUPs on the remaining 13 norms, we found that the mean specificity ratio for every model is larger than 1.0 (Fig. 1 B; range: 1.74–3.43), and exceeds that of the model ratings (range: 1.16–1.49). This indicates that a model’s deviations on one norm are best predicted by its own deviations on other norms rather than by those of other models. The structure of idiosyncrasy varies across norms. Gender Association and Arousal show the strongest variation (34.0% and 22.6%), while Gustatory ratings show the least (9.7%); Morality is distinguished by substantial Bias; and Valence, Socialness, Concreteness, and Age of Acquisition are dominated by shared Trait variance, reflecting strong cross-model consensus. Models correlated only moderately with human norms ( \bar{r}=0.48 – 0.67 ), suggesting LLM semantic representations are partly sui generis rather than noisy approximations of human judgments. Stochastic aggregation consistently outperformed deterministic decoding in predicting human judgments ( \Delta\bar{r}=0.032 ; 10/10 models, 14/14 norms), introducing a reproducibility–alignment tradeoff: Deterministic decoding maximizes replicability but sacrifices both variance structure and human alignment. Conclusion Across 14 psycholinguistic norms, model idiosyncrasy accounts for a substantial share of behavioral variance (4.8–34.0%) beyond shared semantic consensus and systematic

Read the complete text