Quantifying and Mitigating Socially Desirable Responding in LLMs: A Desirability-Matched Graded Forced-Choice Psychometric Study
Kensuke Okada, Yui Furukawa, Kyosuke Bunji · 2026
Read Quantifying and Mitigating Socially Desirable Responding in LLMs: A Desirability-Matched Graded Forced-Choice Psychometric Study by Kensuke Okada, Yui Furukawa, Kyosuke Bunji (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-17262.txt.
Description
Kensuke Okada led a three-author team measuring a quiet bias in language model questionnaires. Self-report inventories now benchmark and audit models for persona, safety, and bias, yet they assume honest answers. In evaluative settings, models may drift toward socially preferred replies, a pattern called socially desirable responding, which skews scores and downstream conclusions. The paper builds a psychometric framework to quantify and curb that drift.
To measure it, the same inventory runs under honest versus fake-good instructions, with the gap scored as a direction-corrected effect size from item response theory latent scores. For mitigation, the team builds a graded forced-choice Big Five inventory from desirability-matched item pairs. Across nine instruction-following models on synthetic personas, Likert formats showed large drift, while the matched forced-choice format cut it sharply while largely preserving persona recovery.
Catalog record
| CREATOR | Kensuke Okada, Yui Furukawa, Kyosuke Bunji |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-17262