What Single-Prompt Accuracy Misses: A Multi-Variant Reliability Audit of Language Models

Ranit Karmakar, Jayita Chatterjee · 2026

Read What Single-Prompt Accuracy Misses: A Multi-Variant Reliability Audit of Language Models by Ranit Karmakar, Jayita Chatterjee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-02038.txt.

Introduction

Ranit Karmakar and Jayita Chatterjee audit language models by varying the prompt, rather than trusting a single prompt score. They study fifteen open-weight models, focusing on ten instruction models across five classification and reasoning tests, each under five prompt variants. For every case they record accuracy, two forms of calibration, parse rate, and spread across prompts.

Evaluation choices alone changed conclusions a lot. One definition change shifted calibration by a large mean amount. Pairing a chain-of-thought prompt with a first-character scorer cut apparent accuracy sharply on one test, and repair steps recovered most of the loss. The audit shows single-prompt accuracy misses reliability failures that matter in practice.

Catalog record

CREATORRanit Karmakar, Jayita Chatterjee
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.02038

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

What Single-Prompt Accuracy Misses: A Multi-Variant Reliability Audit of Language Models Ranit Karmakar, Jayita Chatterjee Abstract Single-prompt accuracy is the dominant way to benchmark language models, but it can miss reliability failures that matter. We evaluate a 15-model open-weight corpus, with the main reliability analyses focused on 10 instruct models across five classification and reasoning benchmarks under five prompt variants each, measuring accuracy, token-probability calibration, verbal-confidence calibration, verbal parse rate, and prompt-perturbation spread for every (model × \times dataset × \times variant) cell. We find three broad results. First, evaluation design can materially change the conclusion. Switching Expected Calibration Error (ECE) token from a raw to a label-set-normalised definition changes per-cell calibration by a mean absolute 0.149. More strikingly, pairing a chain-of-thought prompt with a first-character evaluator on ARC-Challenge reduces apparent accuracy by 72–88% across all five primary models; two independent repair procedures recover 93.8% and 102.7% of the lost performance, indicating an evaluator-side rather than model-side failure. Second, confidence signals are fragile. On MMLU-Pro, every primary model verbally reports confidence substantially above both its accuracy and its token-probability confidence on the same rows, and verbal parse rate can collapse for a single model on a single prompt variant. Third, prompt robustness does not track parameter count reliably. Across 10 instruct models, the correlation between model size and prompt-perturbation spread ranges from − 0.244 -0.244 to 0.474 0.474 across benchmarks. Taken together, these results show that reliability conclusions for small language models depend not only on the model being evaluated, but also on the evaluation pipeline used to measure it. We argue that calibration definitions, evaluator logic, verbal parseability, and prompt robustness should be reported explicitly when making reliability claims. 1 Introduction Small language models (SLMs) in the 1–8B parameter range are increasingly attractive for local assistants, on-device applications, and multi-agent systems where inference cost, latency, or privacy constraints make larger models impractical. In these settings, benchmark accuracy is necessary but incomplete. A deployed model must not only answer correctly on average; it must also expose usable confidence signals, produce outputs that downstream systems can parse, and remain stable under the prompt variants that users or pipelines actually induce. Most SLM evaluations still emphasize single-prompt accuracy. This is useful for ranking models, but it hides several reliability questions that matter in deployment. Does the model remain calibrated when the prompt format changes? Does verbal confidence track correctness, or merely sound confident? Can the evaluator reliably extract the intended answer from the generated text? And does robustness improve with parameter count, as practitioners often assume? These questions are difficult to answer from a single accuracy number, and even harder to diagnose when prompt templates, evaluator logic, and calibration definitions are not reported explicitly. We study these issues through a multi-variant reliability audit of 15 open-weight SLMs from 1B to 8B parameters. We evaluate five classification and reasoning benchmarks under five prompt variants per instance, recording accuracy together with token-probability calibration, verbal-confidence calibration, verbal parse rate, and prompt-perturbation spread for each model–dataset–variant cell. This design lets us observe failures that would be averaged away by model-level or single-prompt reporting. Our results show that reliability conclusions can depend as much on the evaluation pipeline as on the model being evaluated. First, seemingly reasonable evaluation choices can materially change conclusions: label-set normalization changes token-calibration estimates substantially, and pairing chain-of-thought output with a first-character evaluator produces a large apparent ARC-Challenge accuracy collapse that is largely repaired by changing only the evaluator. Second, confidence signals are fragile: on MMLU-Pro, primary models verbally report confidence far above both accuracy and token-probability confidence, while verbal parseability can fail for a single model on a single prompt variant. Third, prompt robustness is not predicted by parameter count in our 1–8B instruct-model sample; smaller models can be substantially more stable than larger ones under prompt perturbation. This paper makes three contributions: • We identify two evaluation-side failure modes—calibration-normalization sensitivity and prompt–evaluator mismatch—that change SLM reliability conclusions without changing the model. • We jointly report verbal-confidence direction and verbal parse rate per model–dataset–variant cell, showing that verbal confidence can be both overconfident and selectively unparseable. • We measure prompt-perturbation spread across 10 instruct models and show that model size is a poor proxy for robustness in this range. Together, these findings argue for reporting SLM reliability as a deployment-readiness profile rather than a single benchmark score: calibration definitions, evaluator logic, verbal parseability, and prompt robustness should be made explicit whenever reliability claims are made. 2 Related Work Calibration and verbal confidence in language models: Expected Calibration Error (ECE) is a standard summary of probabilistic calibration, building on binned reliability estimates 11 ; 5 . In language models, calibration can be measured from token probabilities, from generated verbal confidence, or from post-hoc transformations of either signal. Prior work has shown that larger RLHF-tuned models can sometimes verbalize useful uncertainty estimates 7 ; 15 , while other studies find systematic verbal overconfidence, especially on harder tasks 17 . Related work has also trained models to express uncertainty directly in natural language 9 . Our work focuses on the smaller 1–8B open-weight regime and measures token confidence, verbal confidence, and verbal parseability jointly at the level of each model–dataset–prompt-variant cell, rather than treating calibration as a single model-level property. Prompt sensitivity and evaluation design: Language-model evaluations are sensitive to prompt wording, formatting, and in-context examples. Prior work has shown that prompt-based models may not use instructions in semantically literal ways 16 , that few-shot prompts can introduce output-distribution biases 19 , and that semantically similar or superficially reformatted prompts can produce large accuracy differences 10 ; 12 . Multi-dimensional evaluation frameworks such as HELM 8 , BIG-bench 13 , and the EleutherAI LM Evaluation Harness 3 have made broad, standardized evaluation more practical, with HELM explicitly including robustness and calibration as evaluation axes. However, most prompt-sensitivity studies emphasize accuracy variance, while most benchmark harnesses fix a canonical prompt and evaluator. Our work studies how prompt perturbations interact not only with accuracy, but also with calibration definitions, verbal-confidence parsing, and evaluator logic. Evaluation of small open-weight models: Recent open-weight SLM releases, including Phi, Llama, Gemma, Qwen, SmolLM, and DeepSeek-R1 distillations, are accompanied by extensive benchmark reports 1 ; 4 ; 14 ; 18 ; 2 ; 6 . These reports are valuable for comparing capabilities across standard tasks, but they primarily emphasize single-prompt accuracy, instruction following, safety, and efficiency. Calibration metrics, prompt-variant robustness, verbal-confidence behavior, and evaluator compatibility are rarely co-reported for the same models on the same examples. This paper complements model-release benchmarks by auditing reliability-side quantities across 15 open-weight SLMs, five benchmarks, and five prompt variants, showing that reliability conclusions can change with the evaluation pipeline itself. 3 Methods We study how evaluation design changes the reliability conclusions one would draw about small language models. Our evaluation covers 15 open-weight models in the 1–8B parameter range, with the main claims carried by a primary set of five instruct models and supplementary models used to widen the robustness analysis and exploratory appendices. We evaluate the models on five primary benchmarks: three classification tasks (SST-2, MNLI, AG News) and two reasoning tasks (ARC-Challenge, MMLU-Pro). For each (model, dataset, prompt variant) cell, we evaluate 500 examples.

Read the complete text