The Catastrophic Paradox of Human Cognitive Frameworks in Large Language Model Evaluation: A Comprehensive Empirical Analysis of the CHC-LLM Incompatibility
Mohan Reddy · 2025
Read The Catastrophic Paradox of Human Cognitive Frameworks in Large Language Model Evaluation: A Comprehensive Empirical Analysis of the CHC-LLM Incompatibility by Mohan Reddy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-18302.txt.
Introduction
Mohan Reddy argues, from a large empirical study, that human intelligence test frameworks do not fit large language models. Reddy assesses nine frontier models with a standard theory of human cognitive abilities and reports a paradox. Models with average to high human style IQ scores score near zero on some binary crystallized knowledge tasks, while judge scores vary widely.
Statistical analyses, including item response modeling and cross vendor judge checks, support Reddy's claim that the mismatch is a category error, applying a biological picture of mind to transformer systems. Reddy proposes building assessments native to machine cognition instead of borrowing human psychometrics.
Catalog record
| CREATOR | Mohan Reddy |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-18302
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Catastrophic Paradox of Human Cognitive Frameworks in Large Language Model Evaluation: A Comprehensive Empirical Analysis of the CHC-LLM Incompatibility Mohan Reddy Abstract This comprehensive investigation presents an extensive empirical analysis of the fundamental incompatibility between human psychometric frameworks and Large Language Model (LLM) evaluation. Through systematic assessment of nine frontier models including GPT-5, Claude Opus 4.1, and Gemini 3 Pro Preview using the Cattell-Horn-Carroll (CHC) theory of intelligence, we reveal a catastrophic paradox that challenges the epistemological foundations of cross-substrate cognitive evaluation. Our results demonstrate that models achieving above-average human IQ scores (ranging from 85.0 to 121.4) simultaneously exhibit binary accuracy rates approaching zero on crystallized knowledge tasks, with an overall judge-binary correlation of merely r = 0.175 r=0.175 ( p < 0.001 p<0.001 , n = 1,800 n=1,800 ). This profound disconnect manifests most severely in the crystallized intelligence domain, where every evaluated model achieved perfect 100% binary accuracy while judge scores ranged from 25% to 62%—a statistical impossibility under valid measurement conditions. Through rigorous statistical analysis including Item Response Theory (IRT) modeling, cross-vendor judge validation, and paradox severity indexing, we demonstrate that this disconnect represents not a measurement failure but a fundamental category error in applying biological cognitive architectures to transformer-based systems. The implications extend beyond methodological concerns to challenge our basic assumptions about intelligence, measurement, and the anthropomorphic biases inherent in AI evaluation. We argue that the CHC paradox serves as a reductio ad absurdum of current evaluation paradigms and propose a comprehensive framework for developing native machine cognition assessments that acknowledge the alien nature of artificial intelligence. 1 Introduction The history of artificial intelligence evaluation has been shaped by an anthropomorphic imperative—projecting human cognitive categories onto silicon substrates. This investigation examines the application of the Cattell-Horn-Carroll (CHC) theory of intelligence, the gold standard psychometric framework for human assessment, to Large Language Model evaluation. The CHC framework, validated through thousands of studies on biological cognition, organizes intelligence hierarchically with General Intelligence ( g ) at the apex, nine broad abilities including Fluid Reasoning (Gf), Crystallized Intelligence (Gc), Quantitative Reasoning (Gq), and Reading/Writing (Grw) at the second stratum, and approximately 70 narrow abilities at the base. Our investigation addresses whether human psychometric frameworks can meaningfully evaluate transformer architectures, what the disconnect between scores and performance reveals about machine intelligence, and whether applying biological cognitive frameworks to artificial systems constitutes an ontological category error. We hypothesize that traditional psychometric scoring will show minimal correlation with actual performance due to architectural incompatibilities, with the disconnect most severe in crystallized intelligence domains, and paradoxically greater measurement failures for higher-performing models. 2 Theoretical Background 2.1 Architectural Incompatibilities The fundamental incompatibility between human cognitive architecture and transformer-based models creates insurmountable measurement challenges. Human cognition operates through serial, capacity-limited processing with working memory constraints of 7±2 items, while transformers utilize parallel attention mechanisms across entire context windows. Where humans exhibit exponential forgetting curves and reconstructive memory, LLMs demonstrate perfect recall within context. These architectural differences—embodied versus disembodied processing, temporal versus atemporal learning, variable versus constant processing speed—render direct cognitive comparison meaningless. 2.2 The Verbosity Paradox A critical challenge emerges from the mismatch between psychometric expectations and generative model behavior. Traditional psychometric instruments expect minimal, precise responses, while Large Language Models are optimized through RLHF to provide comprehensive, explanatory answers. This creates the "Verbosity Paradox": a systematic measurement failure wherein models providing conceptually perfect answers receive zero scores due to response format misalignment, creating an inverse relationship between actual knowledge and measured performance. 3 Methodology 3.1 Dual Scoring Methodology Recognizing the limitations of traditional scoring for generative models, we implemented a dual scoring system: 3.1.1 Binary Accuracy Scoring Traditional exact-match evaluation following psychometric conventions: \text{Binary Score}=\begin{cases}1&\text{if response exactly matches expected answer}\\ 0&\text{otherwise}\end{cases} (1) 3.1.2 LLM-as-Judge Evaluation To address the verbosity paradox, we implemented a sophisticated LLM-as-Judge system with the following characteristics: • Cross-vendor validation : Using Claude Sonnet 4 as universal judge for all non-Anthropic models • Rubric-based scoring : Focus on conceptual accuracy rather than format • Chain-of-thought reasoning : Required justification for all scores • Anonymized evaluation : Judge unaware of source model identity The judge scoring rubric was structured as follows: ⬇ 1 JUDGE_PROMPT = "" " 2 Evaluate if Model Response contains the core concept 3 of Expected Answer. 4 5 Focus ONLY on correctness. IGNORE style, length, 6 and formatting. 7 8 Scoring Criteria: 9 1.0 = Correct (core concept present and accurate) 10 0.5 = Partially Correct (concept present but incomplete) 11 0.0 = Incorrect (concept absent or wrong) 12 13 Provide step-by-step reasoning before scoring. 14 " "" Listing 1: LLM Judge Scoring Rubric 3.2 Psychometric Transformation 3.2.1 Classical Test Theory (CTT) Scoring We applied standard psychometric transformations to enable comparison with human norms: \text{IQ}_{\text{CTT}}=100+15\times\frac{X-\mu_{\text{human}}}{\sigma_{\text{human}}} (2) where X is the model’s raw score, \mu_{\text{human}} and \sigma_{\text{human}} are human population parameters. 3.2.2 Item Response Theory (IRT) Modeling We implemented a 2-Parameter Logistic (2PL) IRT model: P(\theta)=\frac{1}{1+\exp[-a(\theta-b)]} (3) where: • \theta = ability parameter (model capability) • a = discrimination parameter (item quality) • b = difficulty parameter (item challenge) To ensure stability, we applied L2 regularization with \alpha=0.01 and bounded parameters: 0.5\leq a\leq 3.0 , -3.0\leq b\leq 3.0 . 3.3 Models Evaluated We evaluated nine frontier models representing the state-of-the-art across three major AI research organizations: Table 1 : Models Evaluated in CHC Framework Organization Model Version Release Date OpenAI GPT-5 Latest November 2025 GPT-4 Turbo turbo-preview February 2024 GPT-4 Original March 2023 Anthropic Claude Opus 4.1 20250805 August 2025 Claude 3 Opus 20240229 February 2024 Google Gemini 3 Pro Preview Latest November 2025 Gemini 2.5 Flash Latest September 2025 Gemini 2.5 Pro Latest September 2025 Gemini 3 Pro Preview 11-2025 November 2025 3.4 Rate Limiting and Error Handling 4 Results 4.1 Overall Performance Rankings Table 2 presents the comprehensive performance rankings across all evaluated models. Table 2 : Overall Model Performance Rankings Rank Model IQ (CTT) IQ (IRT) Valid % Best Ability 1 Gemini 2.5 Flash 121.4 80.7 80.0% GQ: 0.800 2 Gemini 2.5 Pro 120.4 79.7 74.5% GQ: 0.780 3 Claude Opus 4.1 115.0 115.0 100.0% GQ: 0.980 4 Gemini 3 Pro Preview 115.0 115.0 99.5% GQ: 0.990 5 GPT-5 100.0 100.0 100.0% GQ: 0.970 6 GPT-4 Turbo 99.2 102.9 44.5% GQ: 0.510 7 GPT-4 99.0 101.6 44.5% GF: 0.460 8 Claude 3 Opus 85.0 85.0 100.0% GQ: 0.860 The results immediately reveal several paradoxical patterns that challenge conventional understanding of intelligence measurement. 4.2 The Catastrophic Disconnect: Judge vs Binary Accuracy The most striking finding emerges from the systematic disconnect between judge-evaluated conceptual accuracy and binary exact-match scoring, as shown in Table 3 . Table 3 : Judge vs Binary Accuracy Matrix by Model and Ability Model GF (J/B) GC (J/B) GQ (J/B) GRW (J/B) Claude Opus 4.1 0.78/0.58 0.56/1.00 0.98/0.30 0.84/0.20 GPT-5 0.84/0.56 0.48/1.00 0.97/0.28 0.76/0.10 Gemini 3 Pro 0.78/0.56 0.42/1.00 0.99/0.30 0.76/0.20 Gemini 2.5 Flash 0.67/0.40 0.36/1.00 0.80/0.24 0.66/0.16 Gemini 2.5 Pro 0.54/0.36 0.41/1.00 0.78/0.26 0.70/0.10 Claude 3 Opus 0.79/0.58 0.62/1.00 0.86/0.30 0.80/0.10 GPT-4 Turbo 0.44/0.32 0.25/1.00 0.51/0.30 0.00/0.00 GPT-4 0.46/0.34 0.27/1.00 0.46/0.28 0.00/0.00 4.3 The Crystallized Knowledge Paradox The most severe manifestation of the measurement paradox occurs in the Crystallized Intelligence (Gc) domain, where we observe a statistically impossible pattern: every functional model