DICE: A Framework for Dimensional and Contextual Evaluation of Language Models
Aryan Shrivastava, Paula Akemi Aoyagui · 2025
Read DICE: A Framework for Dimensional and Contextual Evaluation of Language Models by Aryan Shrivastava, Paula Akemi Aoyagui (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-10359.txt.
Introduction
Aryan Shrivastava and Paula Akemi Aoyagui propose a 2025 frame for judging language models by the setting in which they will be used. Models now sit inside many tools, yet standard tests often miss how people use them in real work. A score on a fixed test may say little about worth to a team with its own needs and limits.
The frame scores models on fine, split traits rather than one mark. Some traits apply in any setting, such as staying sound under odd inputs, keeping sense, and being frank about doubt. Other traits must be shaped to each setting and to the people who must trust the tool. The paper lays out ways to put the frame to work and the gains and hard parts it brings.
Catalog record
| CREATOR | Aryan Shrivastava, Paula Akemi Aoyagui |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-10359
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DICE: A Framework for Dimensional and Contextual Evaluation of Language Models Aryan Shrivastava, Paula Akemi Aoyagui Abstract . Language models (LMs) are increasingly being integrated into a wide range of applications, yet the modern evaluation paradigm does not sufficiently reflect how they are actually being used. Current evaluations rely on benchmarks that often lack direct applicability to the real-world contexts in which LMs are being deployed. To address this gap, we propose Di mensional and C ontextual E valuation (DICE), an approach that evaluates LMs on granular, context-dependent dimensions. In this position paper, we begin by examining the insufficiency of existing LM benchmarks, highlighting their limited applicability to real-world use cases. Next, we propose a set of granular evaluation parameters that capture dimensions of LM behavior that are more meaningful to stakeholders across a variety of application domains. Specifically, we introduce the concept of context-agnostic parameters—such as robustness, coherence, and epistemic honesty—and context-specific parameters that must be tailored to the specific contextual constraints and demands of stakeholders choosing to deploy LMs into a particular setting. We then discuss potential approaches to operationalize this evaluation framework, finishing with the opportunities and challenges DICE presents to the LM evaluation landscape. Ultimately, this work serves as a practical and approachable starting point for context-specific and stakeholder-relevant evaluation of LMs. 1. Introduction Language models (LMs) have significantly evolved from the early n-gram models first proposed in 1948 ( Shannon, 1948 ) to powerful neural models that are highly capable and general. Furthermore, new training paradigms have enabled models to follow user instructions ( Ouyang et al., 2022 ; Chung et al., 2024 ) , making them more approachable to broader society. This has led to the widespread adoption of LMs across a range of real-world domains. For example, LMs are now applied in healthcare ( Meng et al., 2024 ; Sarraju et al., 2023 ; Yang et al., 2022 ; Thirunavukarasu et al., 2023 ; Singhal et al., 2023 , e.g.,) , education ( Wang et al., 2024 ; Extance, 2023 ; Javaid et al., 2023 ; Copilot, 2024 , e.g.,) , law ( Lai et al., 2024 ; AI, 2025 ; Gainer and Starostin, 2024 ; LexisNexis, 2024 , e.g.,) , finance ( Li et al., 2023a ; Nie et al., 2024 ; Maple et al., 2024 ; Spyrou and Pisaneschi, 2024 , e.g.,) , and human resources ( Xu et al., 2024b ; O’Neal, 2024 ; Xu et al., 2024a ; Ma et al., 2024 , e.g.,) . The proliferation of LMs into diverse domains emphasizes the need for effective, meaningful, and relevant evaluation methods to highlight their context-specific capabilities and limitations. Particularly, stakeholders, including those that are non-technical, rely on LM evaluations to determine, whether, how, and which models to deploy into their specific context. The standard approach to evaluating modern LMs is via benchmarks. They serve as a standardized set of tasks that assess key capabilities and limitations of LMs, facilitating consistent evaluation which ultimately enables comparisons between models under various settings ( McIntosh et al., 2024 ) . As such, benchmark results are often the primary factor stakeholders consider when deciding whether to deploy an LM in a specific context and, if so, which model best meets their needs. However, many works have noted the limitations of the benchmarking paradigm in LM evaluation. For one, benchmarks suffer from construct invalidity, meaning they often do not sufficiently measure what they are intended to measure ( Raji et al., 2021 ) . This issue is exacerbated largely due to overgeneralized claims of LM performance made by model developers ( Reuel et al., 2024 ) . Additionally, most benchmarks measure performance on tasks that are not relevant to how LMs are being used in the real-world ( McIntosh et al., 2024 ) . This highlights the insufficiency of relying solely on benchmarks to guide deployment decisions across contexts. Thus, in this position paper, we propose DICE ( Di mensional & C ontextual E valuation), a framework that granularizes and contextualizes LM evaluation to better support domain-specific stakeholders. We begin by reviewing the current landscape of LM evaluation, considering both the utility and limitations of benchmarks (§ 2 ). Next, we outline DICE (§ 3 ). As a part of this framework, we introduce the concept of context-agnostic dimensions, which are relevant across all domains, and context-specific dimensions, which must be designed to the particular needs and constraints of a given domain. We also discuss how DICE can be operationalized. Finally, we explore the opportunities this framework presents for improving LM evaluation alongside the challenges that may arise with regards to its adoption (§ 5 ). Ultimately, we contribute a novel perspective to LM evaluation with the aim to make LM evaluation more contextual and stakeholder-relevant. This work serves as a precursor to a larger, empirical study and we hope to foster further discussion on the development of more adaptive, context-aware evaluation methodologies that better reflect real-world requirements. 2. Current Evaluations of Language Models 2.1. The Modern Benchmarking Paradigm Currently, LMs are typically evaluated via benchmarks : “a dataset \dots and a metric, conceptualized as representing one or more specific tasks or sets of abilities, picked up by a community of researchers as a shared framework for the comparison of methods” ( Raji et al., 2021 ) . It has become standard practice to use these benchmarks to track progress, identify weaknesses, and facilitate comparative analysis of LMs ( Reuel et al., 2024 ) . For example, model developers use benchmarks to 1) guide the development of models by identifying areas for improvement and validating progress and 2) encourage the adoption of their models by reporting state-of-the-art results. Furthermore, model users often rely on benchmark performance to determine which LM best suits their specific use case—or whether to use one at all. Overall, benchmarking offers a standardized approach to evaluating LMs, serving the needs of developers and users alike. Over time, numerous benchmarks have emerged, evaluating different aspects of LM performance. While there are many valid categorizations, we broadly categorize benchmarks into those of: General-Purpose Factual Knowledge. These benchmarks consist of question datasets with verifiable factual answers, typically formatted as multiple-choice or short-answer tasks. These encompass some of the most widely reported benchmarks in LM evaluation. Prominent examples include GLUE ( Wang et al., 2018 ) , MMLU ( Hendrycks et al., 2021a ) , BIG-bench ( Srivastava et al., 2023 ) , and Humanity’s Last Exam ( Phan et al., 2025 ) . Reasoning Ability. These benchmarks assess an LM’s ability for logical deduction and problem-solving. Mathematical reasoning is a commonly tested area, as shown by MATH ( Hendrycks et al., 2021b ) and GSM8K ( Cobbe et al., 2021 ) . Additionally, DROP ( Dua et al., 2019 ) measures discrete reasoning over multiple paragraphs, while ARC-AGI ( Chollet, 2019 ) measures abstract-reasoning and pattern-recognition. The recent emergence of “reasoning” models such as OpenAI’s o-series ( OpenAI, [n. d.] ) and the DeepSeek models ( DeepSeek, [n. d.] ) underscores the importance of evaluating reasoning ability. Human-Preference Alignment. These benchmarks assess how well an LM’s outputs align with human preferences, extending beyond evaluations of objective correctness. Typically performance is measured via