Human- vs. AI-generated tests: dimensionality and information accuracy in latent trait evaluation

Mario Angelelli, Morena Oliva, Serena Arima, Enrico Ciavolino · 2025

Read Human- vs. AI-generated tests: dimensionality and information accuracy in latent trait evaluation by Mario Angelelli, Morena Oliva, Serena Arima, Enrico Ciavolino (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24739.txt.

Introduction

Mario Angelelli, Morena Oliva, Serena Arima, and Enrico Ciavolino check if AI made survey forms can match a human made one. They took a known form on body sense and had a chat model make two new forms of it. One AI form had more plain cues on what each part should tap than the other.

Though the words looked close to the source, the forms split in key ways. Their shape in terms of traits tapped, and where they gave the most facts across trait levels, both shifted. A Bayes graded model was used to gauge the forms. The lesson is that word feel is not enough; stat checks must back AI forms.

Catalog record

CREATORMario Angelelli, Morena Oliva, Serena Arima, Enrico Ciavolino
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE86 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-24739

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Human- vs. AI-generated tests: dimensionality and information accuracy in latent trait evaluation Mario Angelelli, Morena Oliva, Serena Arima, Enrico Ciavolino Abstract Artificial Intelligence (AI) and large language models (LLMs) are increasingly used in social and psychological research. Among potential applications, LLMs can be used to generate, customise, or adapt measurement instruments. This study presents a preliminary investigation of AI-generated questionnaires by comparing two ChatGPT-based adaptations of the Body Awareness Questionnaire (BAQ) with the validated human-developed version. The AI instruments were designed with different levels of explicitness in content and instructions on construct facets, and their psychometric properties were assessed using a Bayesian Graded Response Model. Results show that although surface wording between AI and original items was similar, differences emerged in dimensionality and in the distribution of item and test information across latent traits. These findings illustrate the importance of applying statistical measures of accuracy to ensure the validity and interpretability of AI-driven tools. Keywords : Latent trait models; Bayesian IRT; Generative AI; ChatGPT; Test information. 1 Introduction The use of artificial intelligence (AI) in data science for social research is gaining increasing attention [ 11 ] . Different AI tools support a variety of functionalities, including data augmentation of evidence from surveys [ 11 ] , data curation, categorisation, and annotation [ 20 ] for thematic assignment [ 19 ] , as well as pattern recognition, clustering, network analysis, and text analysis. Focusing on the latter, textual data are essential in social research for deeper investigations of attitudes and perceptions, and appropriate analysis methods are needed when qualitative and quantitative structures should be extracted from (unstructured) textual data. A substantial stimulus to extracting information and meaningful indicators from textual data is represented by generative AI, especially large language models (LLMs). These types of artificial neural networks include attention mechanisms and appropriate encoding of textual information—i.e., embedding. Chatbots based on LLMs, such as ChatGPT, are widely used in a multi-task setting, which includes text translation and multilingual studies [ 35 ] , summarisation, extraction of analytics and quantitative text analysis (including sentiment analysis), and conceptual analysis. An example where LLMs may be valuable support is also represented by cross-cultural research, where the measurement of social constructs can be affected by linguistic factors. While the capabilities of generative AI are well established for the processing of collected data, the production of new measurement tools, such as questionnaires, tests, and surveys, is an emerging opportunity that stresses the role of interactions between human experts and LLMs. Thanks to an appropriate prompt design, LLMs can produce prototypes of measurement instruments by training the algorithms with a large documental corpus, and they are able to contextualise the extracted information, aligning it with the scope of the research expressed in the prompt. However, the outputs of LLMs should be carefully analysed since they are not the outcome of a consistent modelling of a conceptually grounded theory. In particular, the well-known limitations of current generative AI, which include the lack of explainability, transparency, and the need for unbiased and fair data, become critical when social constructs are investigated. For example, the lack of explainability may limit the replicability of the results, their generalisability, and, hence, the extraction of usable knowledge to support social intervention based on causal effects. Linguistic biases in training data could be reflected in the generated text [ 38 , 30 , 39 ] , with the potential effect of reinforcing stereotypical expressions or concepts [ 1 ] even during the collection of new data. The occurrence of unattainable results [ 2 ] entails the need for human supervision to avoid producing measurements based on inaccurate information. Despite the efficiency of LLMs in translation and cultural adaptation of text, the misuse of language-specific wording may affect the LLM performance [ 7 ] and, in the context of test or scale development, generate misalignment between the original and the translated versions, requiring careful revision by experts. Bias in training data may also be enhanced when new synthetic data are produced and, in turn, used for subsequent training [ 51 ] . Furthermore, research linked to sensitive topics and psychological aspects calls for compliance with methodological and ethical standards and appropriate use of data to preserve their confidentiality and integrity. These issues raise concerns regarding the validity of AI-generated questionnaires, which could compromise the quality of social indicators. This work aims to conduct a preliminary investigation of accuracy criteria and methods for comparing validated instruments for psychological measurement with large language model (LLM)-generated alternatives. In particular, it examines how different information criteria can distinguish human- from AI-generated questionnaires. Such methods can enhance accuracy assessment and guide the adaptation or tailoring of questionnaires by evaluating how well test items capture information about respondents across different scores or latent traits in AI-generated solutions. First, data are analysed in terms of reliability; following an exploratory factor analysis, we focus on the dimensionality of the questionnaires in terms of latent traits needed to explain the response pattern to items, with special attention to essential unidimensionality [ 46 ] . Furthermore, we explore specific qualitative and quantitative properties of the tests, namely, the ranking of items based on their characteristics (difficulty, discrimination power) and the information they convey across multiple values of the latent traits. These properties and their mutual relation uncover differences between human- and AI-generated measurements, despite their facet similarity. Results pave the way for methodological advances in the exploration of ambiguity in AI-generated measurements, namely, unknown or hidden sources of epistemic uncertainty that may arise when moving from knowledge-based and theoretically grounded measurement design to black box approaches [ 5 ] . The rest of the work is organised as follows: in Section 2 , we provide an overview of the state-of-the-art of the use of generative AI, especially LLMs, in social and psychological research. Item response theory and its generalisations, in particular graded response models, are presented in Section 3 . The data collection procedure, including the prompt design, is reported in detail in Section 4 . Section 5 presents the results of the analyses linked to reliability and exploratory factor analysis, while Section 6 concentrates on the dimensionality and information properties of the three questionnaires. Such results are discussed and interpreted in Section 7 . Finally, we draw conclusions and, starting from the limitations of this work, identify future research directions in Section 8 . 2 State of the art Given the novelty of LLM-based tools, the research on their implications in social and psychological research is still in its early stages. To better contextualise the motivations that stimulated this work, we provide an overview of AI applications in the measurement of sociopsychological constructs. 2.1 Generative AI in social research While automated reporting represents one of the most common applications of AI-powered chatbots, recent studies show that LLMs are perceived as useful even in the conceptual development phases, as they can support researchers in structuring complex models [ 34 ] . Evidence of the high performance of LLMs is being collected for a wide range of methodologies, e.g., qualitative analysis of textual data [ 52 ] . Remarkably, LLMs can provide substantial help to domain experts in producing software through assisted coding to support the

Read the complete text