Beyond Benchmark: LLMs Evaluation with an Anthropomorphic and Value-oriented Roadmap

Jun Wang, Ninglun Gu, Kailai Zhang, Zijiao Zhang, Yelun Bao, Jin Yang, Xu Yin, Liwei Liu, Yihuan Liu, Pengyong Li, Gary G. Yen, Junchi Yan · 2025

Read Beyond Benchmark: LLMs Evaluation with an Anthropomorphic and Value-oriented Roadmap by Jun Wang, Ninglun Gu, Kailai Zhang, Zijiao Zhang, Yelun Bao, Jin Yang, Xu Yin, Liwei Liu, Yihuan Liu, Pengyong Li, Gary G. Yen, Junchi Yan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-18646.txt.

Introduction

Jun Wang and ten co-authors argue that AI models need a broader report card in a 2025 paper. Large language models score well on test sets, yet those scores often fail to match real-world use. Current testing is scattered and leans on narrow technical measures.

The authors frame a model's traits in human terms, as four quotients: smarts, job skill, emotional sense, and values. They tie each test area back to the stage of training that shapes it, so a weak score points to a cause, not just a rank. A review of over two hundred public benchmarks supports the plan and maps the main gaps. The team also keeps a public list of evaluation resources. Their goal is a compass for building models that are skilled, fitting, and sound in values. The paper is a roadmap for the field rather than a single new test.

Catalog record

CREATORJun Wang, Ninglun Gu, Kailai Zhang, Zijiao Zhang, Yelun Bao, Jin Yang, Xu Yin, Liwei Liu, Yihuan Liu, Pengyong Li, Gary G. Yen, Junchi Yan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE112 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-18646

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Benchmark: LLMs Evaluation with an Anthropomorphic and Value-oriented Roadmap Jun Wang, Ninglun Gu, Kailai Zhang, Zijiao Zhang, Yelun Bao, Jin Yang, Xu Yin, Liwei Liu, Yihuan Liu, Pengyong Li, Gary G. Yen, Junchi Yan Abstract For Large Language Models (LLMs), a disconnect persists between benchmark performance and real-world utility. Current evaluation frameworks remain fragmented, prioritizing technical metrics while neglecting holistic assessment for deployment. This survey introduces an anthropomorphic evaluation paradigm through the lens of human intelligence, proposing a novel three-dimensional taxonomy: Intelligence Quotient (IQ)-General Intelligence for foundational capacity, Emotional Quotient (EQ)-Alignment Ability for value-based interactions, and Professional Quotient (PQ)-Professional Expertise for specialized proficiency. For practical value, we pioneer a Value-oriented Evaluation (VQ) framework assessing economic viability, social impact, ethical alignment, and environmental sustainability. Our modular architecture integrates six components with an implementation roadmap. Through analysis of 200+ benchmarks, we identify key challenges including dynamic assessment needs and interpretability gaps. It provides actionable guidance for developing LLMs that are technically proficient, contextually relevant, and ethically sound. We maintain a curated repository of open-source evaluation resources at: https://github.com/onejune2018/Awesome-LLM-Eval . Impact Statement —As LLMs rapidly transition from research prototypes to real-world applications, the field faces a fundamental disconnect between benchmark performance and practical utility. Current evaluation practices remain fragmented, prioritizing isolated technical metrics while neglecting the developmental trajectory of LLM capabilities and their broader societal implications. This review addresses this critical gap by introducing an anthropomorphic evaluation paradigm that maps LLM assessment to human cognitive progression, through a novel four-dimensional IQ-EQ-PQ-VQ taxonomy. Crucially, our framework establishes the first comprehensive roadmap that reveals how evaluation dimensions correspond to LLMs’ developmental stages: IQ (pre-training knowledge acquisition), PQ (supervised fine-tuning expertise), EQ (reinforcement alignment), and VQ (value-oriented impact). This work provides not merely an assessment tool but a strategic compass for navigating the rapidly evolving landscape of AI evaluation. The roadmap enables stakeholders to anticipate future challenges while selecting context-appropriate evaluation strategies across the model lifecycle. I Introduction The quest to understand intelligence, particularly human intelligence, has been a long-standing pursuit. Throughout history, humans have employed various methods to measure and evaluate cognitive abilities, from traditional IQ tests and cognitive games to more complex assessments through education and professional achievements. This ongoing exploration aims to define, assess, and expand the boundaries of human intellect [ 1 ] . Contemporarily, the rise of machine intelligence, especially LLMs within natural language processing (NLP), has introduced a new dimension to this inquiry [ 2 , 3 ] . These LLMs show remarkable capabilities in understanding and generating language, thereby prompting a critical need for effective measures and evaluation frameworks to gauge their level with respect to human intelligence [ 4 , 5 ] . Formerly, the NLP community relied on simple benchmark tests to evaluate language models, focusing primarily on aspects like grammar and vocabulary. As the field progressed, more sophisticated benchmarks emerged, such as the MUC evaluations [ 6 ] , which concentrated on information extraction. With the advent of deep learning, the landscape further evolved, incorporating comprehensive benchmarks like SNLI [ 7 ] , SQuAD [ 8 ] and DROP [ 9 ] , which not only assessed performance but also provided substantial training data. Particularly, the emergence of large-scale pre-trained language models, such as BERT [ 2 ] , marked a paradigm shift, necessitating the development of new evaluation methodologies. This led to a proliferation of shared tasks and challenges, including SemEval [ 10 ] , CoNLL [ 11 ] , GLUE [ 12 ] , SuperGLUE [ 13 ] , and XNLI [ 14 ] . These initiatives facilitated a holistic assessment of model performance, fostering continuous improvement in evaluation techniques. As LLMs have grown in size and capability, they have demonstrated impressive performance in both zero-shot and few-shot scenarios, often rivaling fine-tuned models [ 1 ] . This has led to a transition from task-specific benchmarks to more general capability assessments, blurring the lines between distinct downstream applications. The rising benchmarks are designed to evaluate a wide range of abilities without relying on extensive training data, thus providing a more comprehensive evaluation under limited-shot conditions [ 15 , 16 , 12 ] . There is a need for rigorous and multifaceted evaluations not only assessing the capabilities but also ensuring alignment with human values and preferences. Pinpointing the limitations in existing evaluation techniques and devising approaches to overcome these hurdles is crucial. Nevertheless, the evaluation of LLMs is a multifaceted and resource - demanding endeavor, encompassing numerous dimensions and facets. Several recent reviews [ 17 , 18 ] have examined the assessment of LLMs, yet their focus has been predominantly on benchmark tasks, datasets, and evaluation metrics, with a lack of in-depth investigation. Such an omission may compromise the validity of the evaluation process, as it overlooks crucial aspects such as practical applicability and interpretability. In an effort to bridge this gap, this paper integrates practical discourse to tackle the foundational challenges and limitations inherent in LLM evaluations that arise from varied evaluation configurations. {forest} Fig. 1: Overview of contents of this paper (zoom in). Recent efforts have proposed taxonomies for evaluating LLMs. Specifically, [ 17 ] categorizes evaluations into knowledge, alignment, and safety, and [ 19 ] focuses on general taxonomies that prioritize abstract categorization, these frameworks often lack granularity in addressing domain-specific proficiency and human-centric practicality. Noteworthily, [ 20 ] reveal that the sampling mechanism of LLMs in decision-making exhibits a descriptive and prescriptive pattern akin to human. This enlightens an anthropomorphic perspective, allowing for a more intuitive and comprehensive assessment across scenarios. As a potential road map to address these limitations, we observe a profound correspondence between LLM evaluation dimensions and the model’s developmental trajectory that mirrors human cognitive progression. As shown in Fig. 2 , the proposed anthropomorphic framework naturally emerges from the three-stage training paradigm defining modern LLM development: Intelligence Quotient (IQ)-General Intelligence : Corresponds to capabilities developed during pre-training , where models acquire foundational knowledge through self-supervised learning on massive corpora. IQ quantifies reasoning ability and world knowledge breadth, analogous to human cognitive foundations. Professional Quotient (PQ)-Professional Expertise : Emerges from supervised fine-tuning (SFT) , where models develop task-specific proficiency through instruction-response pairs. PQ measures specialized capabilities across diverse application domains. Emotional Quotient (EQ)-Alignment Ability : Cultivated through post-training reinforcement learning (RL) , where models learn to align outputs with human values. EQ assesses emotional and ethical resonance with human preferences beyond mere task completion. Unlike broad ’knowledge’ dimension in [ 19 ] , our IQ evaluation explicitly quantifies foundational reasoning and world knowledge breadth, while PQ introduces a structured evaluation of task-specific expertise, which existing frameworks neglect. Furthermore, EQ extends beyond [ 17 ] ’s safety-centric alignment to encompass emotional and ethical alignment with human values, ensuring outputs resonate with user preferences and societal norms. Early stages of LLM evaluation mainly focus on IQ, ensuring that the models had a broad base of world knowledge. As pre-training techniques and data engineering matured, the emphasis shifted to PQ, evaluating the model’s ability to solve specific practical tasks. Now, as models become proficient in these tasks, EQ has become increasingly important. For IQ and PQ, there are well-established benchmarks such as MMLU [ 16 ] , GPQA [

Read the complete text