CAT: A Metric-Driven Framework for Analyzing the Consistency-Accuracy Relation of LLMs under Controlled Input Variations

Paulo Cavalin, Cassia Sanctos, Marcelo Grave, Claudio Pinhanez, Yago Primerano · 2025

Read CAT: A Metric-Driven Framework for Analyzing the Consistency-Accuracy Relation of LLMs under Controlled Input Variations by Paulo Cavalin, Cassia Sanctos, Marcelo Grave, Claudio Pinhanez, Yago Primerano (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-23711.txt.

Introduction

Paulo Cavalin and four co-authors introduce CAT, a framework for judging language models on two virtues at once: being right and being consistent. A model can score well on a benchmark yet change its answer when the question is slightly reworded, a serious flaw in high-stakes use. Current evaluations report accuracy, and sometimes consistency, but rarely study how the two trade off.

CAT varies model inputs in controlled ways and plots consistency-accuracy relation curves, showing how accuracy changes as the required consistency threshold rises, using multiple-choice benchmarks as the test case. A summary index, CORE, compresses each curve's area and shape into one number for the overall trade-off. Demonstrations span generalist and domain-specific models. The authors also sketch how the framework can extend to open-ended, long-form answers with adaptable scoring.

Catalog record

CREATORPaulo Cavalin, Cassia Sanctos, Marcelo Grave, Claudio Pinhanez, Yago Primerano
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-23711

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CAT: A Metric-Driven Framework for Analyzing the Consistency-Accuracy Relation of LLMs under Controlled Input Variations Paulo Cavalin, Cassia Sanctos, Marcelo Grave, Claudio Pinhanez, Yago Primerano Abstract We introduce CAT , a framework designed to evaluate and visualize the interplay of accuracy and response consistency of Large Language Models (LLMs) under controllable input variations, using multiple-choice (MC) benchmarks as a case study. Current evaluation practices primarily focus on model capabilities such as accuracy or benchmark scores and, more recently, measuring consistency is being considered an essential property for deploying LLMs in high-stake, real-world applications. We argue in this paper that although both dimensions should still be evaluated independently, their inter-dependency also need to be considered for a more nuanced evaluation of LLMs. At the core of CAT are the Consistency-Accuracy Relation (CAR) curves, which visualize how model accuracy varies with increasing consistency requirements, as defined by the Minimum-Consistency Accuracy (MCA) metric. We further propose the Consistency-Oriented Robustness Estimate (CORE) index, a global metric that combines the area and shape of the CAR curve to quantify the trade-off between accuracy and consistency. We present a practical demonstration of our framework across a diverse set of generalist and domain-specific LLMs, evaluated on multiple MC benchmarks. We also outline how CAT can be extended beyond MC tasks to support long-form, open-ended evaluations through adaptable scoring functions. 1 Introduction Large Language Models (LLMs) have advanced rapidly in recent years, yet systematic evaluation practices seem to have kept pace. Most evaluation practices have focused on functional capabilities such as benchmark scores or task-specific accuracy while overlooking important non-functional properties [ Aaron Grattafiori et al 2024 , DeepSeek-AI and Aixin Liu et al 2024 , Petrov et al. 2025 ] . One such property is response consistency , considered here as the ability of the LLM to answer similar questions (or prompts , in current LLM-related jargon), which have the same correct answer, with answers similar to the correct answer. This is clearly an essential requirement for deploying LLMs in real-world, high-stake domains like healthcare, law, and finance, where input variations should not lead to inconsistent or contradictory outputs. It is reasonable to expect that, depending on the task and the context, the prevalence of accuracy over consistency, or vice-versa, changes. However, there is a lack of tools and methods to allow developers and system deployment decision-makers to understand the nature of this trade-off for a given system. The framework described in this paper was created to address this methodological gap. We thus present CAT (Consistency-Accuracy Toolkit), a comprehensive framework for evaluating the interaction between response consistency of LLMs and accuracy under input perturbations, with an initial focus on multiple-choice (MC) benchmarks. The core of the framework is the Consistency-Accuracy Relation (CAR) curve, which captures how model accuracy varies as we progressively increase the threshold for acceptable response consistency. This is formalized through the Minimum-Consistency Accuracy (MCA) metric, which defines accuracy conditioned on a minimum level of required consistency across input variants. To summarize the overall trade-off between consistency and accuracy, we introduce the Consistency-Oriented Robustness Estimate (CORE) index, a global metric that combines the area under the CAR curve with its deviation from an idealized perfectly consistent model. Together, our metrics provide a testbed to visualize and measure the interplay between accuracy and consistency, which is key for generative AI applications in real-world. To facilitate the understanding of our approach, we consider in this paper only multiple-choice (MC) benchmarks which have emerged as a valuable and controllable way for studying the accuracy and consistency of LLMs. Their structured format allows researchers to systematically introduce meaning-equivalent input variations, such as simple changes in prompt phrasing, choice order, and formatting, and observe how these perturbations affect model behavior. Leveraging this setup, recent studies have revealed that LLMs are often highly sensitive to even minor changes in input structure [ Habba et al. 2025 , Wang et al. 2025 , Nalbandyan et al. 2025 ] . Notice that we do not explore in this paper issues related to inference consistency , where the ability to answer, under repeated trials, the exact same question or prompt with meaning-equivalent answers, such as examined in [ Pinhanez et al. 2025 ] , but our framework can easily handle such cases. (a) 2 choices (b) 5 choices (c) 10 choices Figure 1: CAR curves for synthetic models starting from a chance model to models with increasing bias toward the correct answer. (a) 2 choices (b) 5 choices (c) 10 choices Figure 2: Metric value growth across synthetic models. CORE emphasizes early gains and reduces bias towards chance. To provide an initial understanding of the framework proposed here, let us consider how three synthetic MC random-answering models of 2, 5, and 10 choices would be represented as CAR (Consistency-Accuracy Relation) curves and their MCA metric values. Figure 1 presents the CAR curves for the three synthetic random-answering experiments. We simulated models with varying levels of positive bias toward the correct answer, i.e. progressive levels of accuracy, and included a baseline chance model (marked in red) as a starting point, where all alternatives have the same probability of being sampled. We then introduced incremental biases of 0.1 in the probability of the correct alternative until it reaches 1.0. Each experiment varied in the number of choices, i.e. 2, 5, and 10, to highlight how the consistency-accuracy relationship evolves with task complexity. These curves reveal a non-linear relationship between consistency and accuracy, which becomes more pronounced as the number of choices increases. Similarly, figure 2 shows how our proposed CORE index captures this non-linearity effectively, unlike two often used accuracy metrics: MCQA+ [ Pezeshkpour and Hruschka 2024 ] which behaves almost linearly as model accuracy increases; and MV [ Wang et al. 2024 ] that tends to saturate quickly, in a behavior which is close to MCA(0.5), that computes accuracy with a very permissive consistency level. Notice that CORE offers a balance between the permissiveness of MCA(0.5) and the strictness MCA(1.0), providing realistic but not overly-punishing scores for the consistency-accuracy interplay. Additionally, CORE reduces bias towards chance-level performance, a limitation often overlooked in existing benchmark metrics. We define these metrics in detail in Section 3 . We present a practical demonstration of CAT on eight LLMs across four MC benchmarks, including both generalist and domain-specific models. Our results show that our approaches provide a complementary and more nuanced view of model behavior, enabling better-informed model selection. We also discuss how the framework can be extended to long-form and open-ended tasks through adaptable scoring functions. The main contributions of this work are: • CAT Framework: a novel evaluation framework designed to assess the interplay between accuracy and consistency of LLM responses, particularly in multiple-choice (MC) settings. This addresses a critical and underexplored non-functional property of LLMs. To support open science, we are releasing the code as open-source at: https://github.com/IBM/cat . • CAR Curves, MCA Metric, and CORE Index: a set of consistency-augmented metrics and tools: the Consistency-Accuracy Relation (CAR) curves leverage the Minimum-Consistency Accuracy (MCA) metric to quantify accuracy under varying consistency thresholds; and the Consistency-Oriented Robustness Estimate (CORE) index, a global metric which summarizes the trade-off between consistency and accuracy. • Practical Demonstration with Models and

Read the complete text