Quantifying and Mitigating Socially Desirable Responding in LLMs: A Desirability-Matched Graded Forced-Choice Psychometric Study

Kensuke Okada, Yui Furukawa, Kyosuke Bunji · 2026

Read Quantifying and Mitigating Socially Desirable Responding in LLMs: A Desirability-Matched Graded Forced-Choice Psychometric Study by Kensuke Okada, Yui Furukawa, Kyosuke Bunji (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-17262.txt.

Description

Kensuke Okada led a three-author team measuring a quiet bias in language model questionnaires. Self-report inventories now benchmark and audit models for persona, safety, and bias, yet they assume honest answers. In evaluative settings, models may drift toward socially preferred replies, a pattern called socially desirable responding, which skews scores and downstream conclusions. The paper builds a psychometric framework to quantify and curb that drift.

To measure it, the same inventory runs under honest versus fake-good instructions, with the gap scored as a direction-corrected effect size from item response theory latent scores. For mitigation, the team builds a graded forced-choice Big Five inventory from desirability-matched item pairs. Across nine instruction-following models on synthetic personas, Likert formats showed large drift, while the matched forced-choice format cut it sharply while largely preserving persona recovery.

Catalog record

CREATORKensuke Okada, Yui Furukawa, Kyosuke Bunji
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-17262

Browse the full library · About the Tower of Babel