Neural network embeddings recover value dimensions from psychometric survey items on par with human data

Max Pellert, Clemens M. Lechner, Indira Sen, Markus Strohmaier · 2025

Read Neural network embeddings recover value dimensions from psychometric survey items on par with human data by Max Pellert, Clemens M. Lechner, Indira Sen, Markus Strohmaier (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24906.txt.

Introduction

Max Pellert, Clemens Lechner, Indira Sen, and Markus Strohmaier wrote a paper in two thousand twenty-five about whether language model embeddings can recover human value dimensions. Psychometric surveys map values such as tradition or self-direction through long questionnaires. The team asked whether the meaning of the questions alone, as read by a neural network, holds the same structure.

Their method, called SQuID, compares embeddings of survey items to extract dimension relations. It recovered value dimensions on par with human rater data, without task-specific training. Its maps aligned with pooled human data from forty-nine countries. Tests on three personality inventories also improved, which hints the method may serve social science more broadly at lower cost.

Catalog record

CREATORMax Pellert, Clemens M. Lechner, Indira Sen, Markus Strohmaier
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-24906

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Neural network embeddings recover value dimensions from psychometric survey items on par with human data Max Pellert, Clemens M. Lechner, Indira Sen, Markus Strohmaier Abstract This study introduces "Survey and Questionnaire Item Embeddings Differentials" (SQuID), a novel methodological approach that enables neural network embeddings to effectively recover latent dimensions from psychometric survey items. We demonstrate that embeddings derived from large language models, when processed with SQuID, can recover the structure of human values obtained from human rater judgments on the Revised Portrait Value Questionnaire (PVQ-RR). Our experimental validation compares multiple embedding models across a number of evaluation metrics. Unlike previous approaches, SQuID successfully addresses the challenge of obtaining negative correlations between dimensions without requiring domain-specific fine-tuning. Quantitative analysis reveals that our embedding-based approach explains 55% of variance in dimension-dimension similarities compared to human data. Multidimensional scaling configurations from both types of data show fair factor congruence coefficients and largely follow the underlying theory. These results demonstrate that semantic embeddings can effectively replicate psychometric structures previously established through extensive human surveys. The approach offers substantial advantages in cost, scalability and flexibility while maintaining comparable quality to traditional methods. Our findings have significant implications for psychometrics and social science research, providing a complementary methodology that could expand the scope of human behavior and experience represented in measurement tools. 1 Introduction Psychometrics, the field of study in psychology that is crucially concerned with measurement of human attributes such as skills, personality, or values, is undergoing a revolution with the integration of novel NLP technologies to complement and extend traditional questionnaire-based workflows. Proposed ideas range from the assessment of large language models via "AI Psychometrics" Pellert et al. 2024 to "Machine Psychology" Hagendorff 2023 or, more conservatively, to use LLMs as assisting tools in human psychometrics to aid comparably cheap, in-silico questionnaire item creation Götz et al. 2023 or automated coding Nilsson et al. 2025 . For the closely related task of synthetic data generation Hommel et al. 2022 , LLMs have been used to simulate segments of the population to match survey results Argyle et al. 2023 over time Ahnert et al. 2025 and to replicate experimental studies Aher et al. 2023 ; Dillion et al. 2023 . The social and behavioral sciences crucially depend on texts that have been rated by human respondents. In psychology this is the traditional approach to analyze traits that are not directly observable such as value orientations Schwartz and Cieciuch 2022 , personality John et al. 2008 and attitudes towards issues like gender beliefs Kerr and Holden 1996 ; Hill and Willoughby 2005 ; Schudson and van Anders 2022 , but the same holds also for fields like political science, sociology and human-computer interaction. Questionnaires, the instruments used to collect human rater judgments for measurement, are typically developed iteratively. First, researchers compile a large set of candidate items based on theories of human experience and behavior. Items are short statements or questions believed to measure an underlying phenomenon, which humans rate on verbal and/or numeric response scales, such as a five-point agree–disagree scale. Through iterative procedures of elimination and refinement, researchers ultimately arrive at a final set of questionnaire items in which each item ideally measures only one underlying dimension of the phenomenon of interest (a property called "simple structure"). In this process, descriptive statistics and correlations among items obtained from samples of human raters are of paramount interest Repke et al. 2024 . Especially the patterns of inter-correlations are crucial in identifying underlying dimensions measured by the items. However, obtaining sufficiently large samples of human raters, and letting them rate a large number of candidate items, is often costly and plagued by issues such as respondent fatigue, response style, low literacy or lack of motivation. These problems can compromise data quality and thwart a proper assessment of the questionnaire’s properties. We argue that addressing the challenge of obtaining fast, cheap and ideally unbiased information on the structure of questionnaires is a promising application area for NLP research. Although the potential is large, critical evaluation has shown various limitations of existing attempts to marry NLP technologies with social scientific applications such as psychometric measurement Löhn et al. 2024 ; Sühr et al. 2024 . We review selected representative approaches by other researchers and discuss the problems they face. Then, we present our novel methodological advancement of "Survey and Questionnaire Item Embeddings Differentials" (SQuID) and demonstrate with the Revised Portrait Value Questionnaire (PVQ-RR) how it allow us to overcome issues that we identified. Research objective: We want to demonstrate the power of neural embeddings in psychometric workflows with the example of the PVQ-RR. Furthermore, we want to argue for this approach to provide useful and interesting complements to human rater judgments. Approach: We use LLMs to create semantic embeddings of the text of psychometric questionnaire items. We choose a widely used survey on value orientations that was carefully developed and extensively validated to compare our embeddings approach to traditional human rater judgments Schwartz and Cieciuch 2022 . Results and contributions: We show that, unlike some of the related works we review claim, semantic embeddings from a general-purpose model in combination with our methodological advancement can indeed recover very well the latent dimensions of a social scientific questionnaire. According to conventionally used quantitative evaluation metrics, we achieve a performance that is at least on par with methods relying on human rater judgments. Our robust findings demonstrate the feasibility and the potential of the "semantic embeddings as human rater judgments" approach and open a large domain of interesting research questions that can be tackled in powerful novel ways by adapting well-established classical psychometric analysis pipelines. While maintaining the same level of quality, our approach based on semantic embeddings is highly flexible, cheap and scales very well. 2 Related Work Researchers have proposed various ways of integrating LLMs in social scientific workflows. Typically, these efforts rely on repeatedly gathering stochastic model output in web interfaces of commercial, closed black-box APIs or on using older, smaller open models Binz and Schulz 2023 ; Fischer et al. 2023 ; Serapio-García et al. 2025 . In this section, we review the most relevant recent literature that we found to be so far in the domain of personality questionnaires. First uses of embeddings to test the structure of psychometric inventories A number of recent studies have proposed using embeddings of questionnaire items from psychometric inventories—or various aggregates of items. These embedding vectors can then be used to create similarity matrices consisting of pairwise similarities between all items or item aggregates. These, in turn, can be analyzed with factor-analytic or similar dimension-reduction methods in much the same way that researchers have analyzed correlation matrices resulting from human subjects’ responses to psychometric inventories. On a fundamental level that also ensures full reproducibility, Cutler and Condon 2023 successfully replicated a classical human subject study on personality traits using an embedding approach. The authors created templates: "Those close to me say I have a [MASK][MASK] and [TERM] personality." [TERM] is inserted from a list of personality descriptors containing entries such as "condescending", the constructed statement is passed through the model and the token embeddings of the two [MASK] tokens are extracted and averaged. The results of principal component analysis

Read the complete text