Measuring How LLMs Internalize Human Psychological Concepts: A preliminary analysis
Hiro Taiyo Hamada, Ippei Fujisawa, Genji Kawakita, Yuki Yamada · 2025
Read Measuring How LLMs Internalize Human Psychological Concepts: A preliminary analysis by Hiro Taiyo Hamada, Ippei Fujisawa, Genji Kawakita, Yuki Yamada (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-23055.txt.
Introduction
Hiro Taiyo Hamada, Ippei Fujisawa, Genji Kawakita, and Yuki Yamada test whether large language models organize psychological ideas the way people do. Their 2025 study notes that models like ChatGPT produce strikingly human-like text, yet how far they truly grasp the concepts behind human thought and behavior was unclear. The team turned to forty-three standard psychological questionnaires with known structure.
Models were asked to compare questionnaire items pairwise, and the resulting similarity clusters were matched against the official categories. GPT-4 sorted items with about sixty-six percent accuracy, ahead of GPT-3.5 and BERT, and well above chance. Its similarity scores also tracked real human response patterns. The framework can probe where model concepts diverge from human ones.
Catalog record
| CREATOR | Hiro Taiyo Hamada, Ippei Fujisawa, Genji Kawakita, Yuki Yamada |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-23055
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Measuring How LLMs Internalize Human Psychological Concepts: A preliminary analysis Thanks: Citation : Authors. Title. Pages…. DOI:000000/11111. Hiro Taiyo Hamada, Ippei Fujisawa, Genji Kawakita, Yuki Yamada Abstract Large Language Models (LLMs) such as ChatGPT have shown remarkable abilities in producing human-like text. However, it is unclear how accurately these models internalize concepts that shape human thought and behavior. Here, we developed a quantitative framework to assess concept alignment between LLMs and human psychological dimensions using 43 standardized psychological questionnaires, selected for their established validity in measuring distinct psychological constructs. Our method evaluates how accurately language models reconstruct and classify questionnaire items through pairwise similarity analysis. We compared resulting cluster structures with the original categorical labels using hierarchical clustering. A GPT-4 model achieved superior classification accuracy (66.2%), significantly outperforming GPT-3.5 (55.9%) and BERT (48.1%), all exceeding random baseline performance (31.9%). We also demonstrated that the estimated semantic similarity from GPT-4 is associated with Pearson’s correlation coefficients of human responses in multiple psychological questionnaires. This framework provides a novel approach to evaluate the alignment of the human-LLM concept and identify potential representational biases. Our findings demonstrate that modern LLMs can approximate human psychological constructs with measurable accuracy, offering insights for developing more interpretable AI systems. K eywords Psychology \cdot Large Language Model \cdot Psychological Concepts 1 Introduction AI systems rapidly accelerate research and development. Applications include automating drug discovery experiments [ 1 ] , exploring medical applications [ 2 ] , and developing alternative sampling and intervention methods in psychology [ 3 ] . Psychological research examines whether Large Language Models (LLMs) can diagnose personality traits [ 4 ] , substitute participant responses [ 5 ] , and mimic fictional character responses. Some studies also showed that LLMs imitate human sensory modalities and cognitive biases [ 6 , 7 ] . These studies suggest that LLMs like GPT-4, trained from vast text data, may learn relationships between psychological concepts and related human linguistic patterns. Measuring human concept internalization in AI systems is important since AI systems may have different concept representations that humans have. It is also known that even representations of human concepts may have different semantics in different cultures [ 8 ] . By understanding different systems like human cultures, we may have better communications with AI systems. This human-AI alignment at the concept level is called concept alignment [ 9 ] . Analyzing them gives us clues on how AI systems internalize concept representations. However, it remains unclear whether language models internalize such concepts. Here, we proposed a novel method to study whether language models, including GPT-4, can internalize conceptual category classifications from psychological questionnaire items. We first compared model classification performance by calculating similarities between items from 43 psychological questionnaires using language models like GPT-3.5 and GPT-4, and measured agreement between similarity-based classifications and original construct labels. We found that GPT-4 showed the highest classification score among other language models while the OpenAI embedding model was comparable. Then, we checked if continuous and discrete evaluations with prompts have different classification performances. The results did not show statistical difference in classification accuracy. We also checked if there is ordering effect of questionnaire items on classification performance. Some questionnaires showed different classification performances. We finally showed that estimated semantic similarity from multiple questionnaires is linked with Pearson’s correlation coefficients of human responses to the questionnaires. Our findings suggest that LLMs retain relationships between psychological concepts. Our framework provides a new tool to measure the level of concept alignment of AI systems. 2 Related Works 2.1 Replicating Human Knowledge and Personality Traits with Language Models Due to the emergence of LLMs, multiple studies explored their ability to replicate human knowledge and personality traits across various domains [ 3 , 10 , 4 ] . In psychology, studies showed reflection of population responses in LLMs. Some studies attempt to imitate human groups and responses of fictional characters [ 11 , 12 ] . [ 12 ] used GPT-3 to answer demographic questions using subject’s demographic information, and found statistically significant correlations between human samples and GPT-3’s responses [ 12 ] . [ 11 ] trained models on anime character dialogues to complete personality questionnaires [ 11 ] . [ 13 ] found association between between human Big Five responses and adjective similarity ratings of the language models [ 13 ] . Other studies also examine whether learned sensory relationships of LLMs reflect human cognitive structures [ 6 , 14 ] . [ 6 ] showed color similarity structure of neuro-typical subjects is more aligned with that of GPT-4 than that of GPT-3.5 [ 6 ] . These studies analyze LLMs which were trained from vast text and image data using psychological testing methods. The correspondence between human and LLM responses suggests the possibility of obtaining equivalent information through LLMs without direct human experiments. This framework aims to understand the responses of AI systems in cognitive tasks, also known as machine psychology [ 15 ] . In this work, we focus on a framework to measure how LLMs internalize psychological dimensions from standardized questionnaires. 3 METHOD 3.1 Database From the psychological questionnaire database "Psychological Scales" ("https://scales.arabpsychology.com/"), we extracted 43 English psychological questionnaires contain 30 or fewer text-based items. Each questionnaire is developed based on psychological constructs such as "curiosity," "self-efficacy," and "anxiety," and includes between 2 and 6 sub-categories labeled by experts. 3.2 Pairwise similarity analysis for classification We compare the classification performance of six language models including bert-base-uncased, Open AI Embedding, gpt-3.5-turbo-0613, gpt-3.5-turbo-0613, gpt-4-turbo-0613 , and gpt-4-turbo-1106-preview on questionnaire items (Table. 1 ). For BERT and an embedding model, we embed all questions from each questionnaire and calculate cosine similarity between items as a similarity matrix (Fig. 1 ). For GPT-3.5 and GPT-4, we provide prompts instructing them to respond within the [-1, 1] range based on semantic similarity between questionnaire items (Table. 2 ). Here, 1 indicates that two items mean identical while -1 indicates that two items mean opposite. The 0 score indicates no similarity between the items. The responses form similarity matrices for each questionnaire. Finally, we apply hierarchical clustering to the generated similarity matrices, using the predetermined number of categories for each questionnaire. For text classification metrics, we use the classification accuracy between classified items and their true categories, along with the adjusted rand index (ARI). Figure 1: Classification with pairwise similarity analysis. 1. A distance between two items from each psychological questionnaire is calculated with language models from -1 to 1. 2. Each similarity matrix was classified with hierarchical clustering with predetermined number of categories for each questionnaire. 3. The labels with unsupervised clustering were compared with true labels with to calculate classification accuracy. ID Questionnaire Name Abbreviation 1 bert-base-uncased (BERT) Parameters: 110M 2 OpenAI embedding An embedding model provided by OpenAI "text-embedding-ada-002" 3 gpt-3.5-turbo-0613 (gpt-3.5-0613) OpenAI API specifications: Parameters: Not disclosed, Context window: 4,096 tokens, Training data: Up to September 2021 4 gpt-3.5-turbo-1106 (pt-3.5-1106) OpenAI API specifications: Parameters: Not disclosed, Context window: 16,385 tokens, Training data: Up to September 2021 5 gpt-4-turbo-0613 (gpt-4-0613) OpenAI API specifications: Parameters: Not disclosed, Context window: 8,192 tokens, Training data: Up to September 2021 6 gpt-4-turbo-1106-preview (gpt-4-1106) OpenAI API specifications: Parameters: Not disclosed, Context window: 128,000 tokens, Training data: Up to