Confidence in the Reasoning of Large Language Models
Yudi Pawitan, Chris Holmes · 2024
Read Confidence in the Reasoning of Large Language Models by Yudi Pawitan, Chris Holmes (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15296.txt.
Introduction
Yudi Pawitan and Chris Holmes measure how much confidence large language models place in their own reasoning. Their 2024 paper tests three models, two from the GPT-4 family and one Mistral model, on causal judgment questions, formal fallacies, and probability puzzles. Reasoning skill in these models is widely studied; the trust they place in their answers is not.
Confidence was probed two ways: whether a model sticks to its answer when asked to reconsider, and the score it reports for itself. All three beat random guessing, yet their stubbornness varied widely. Sticking with an answer did track accuracy, but second answers were often worse than first ones. Self-reported scores ran far too high. Token-level math explained only part of the pattern. The authors conclude that today's models hold no coherent inner sense of confidence.
Catalog record
| CREATOR | Yudi Pawitan, Chris Holmes |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-15296
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Confidence in the Reasoning of Large Language Models Yudi Pawitan, Chris Holmes Abstract There is a growing literature on reasoning by large language models (LLMs), but the discussion on the uncertainty in their responses is still lacking. Our aim is to assess the extent of confidence that LLMs have in their answers and how it correlates with accuracy. Confidence is measured (i) qualitatively in terms of persistence in keeping their answer when prompted to reconsider, and (ii) quantitatively in terms of self-reported confidence score. We investigate the performance of three LLMs – GPT4o, GPT4-turbo and Mistral – on two benchmark sets of questions on causal judgement and formal fallacies and a set of probability and statistical puzzles and paradoxes. Although the LLMs show significantly better performance than random guessing, there is a wide variability in their tendency to change their initial answers. There is a positive correlation between qualitative confidence and accuracy, but the overall accuracy for the second answer is often worse than for the first answer. There is a strong tendency to overstate the self-reported confidence score. Confidence is only partially explained by the underlying token-level probability. The material effects of prompting on qualitative confidence and the strong tendency for overconfidence indicate that current LLMs do not have any internally coherent sense of confidence. 1 Introduction and Summary The emergence of large language models (LLMs) such as OpenAI’s GPT series has sparked significant interest and debate within the field of artificial intelligence. These complex neural-network models, designed as a next-word (technically next-token) predictor and trained on vast amounts of text data, have demonstrated an unprecedented ability to generate coherent and contextually appropriate text responses. This human-like capability has led to speculation about emergent qualities, whether these models can ‘reason’ and ‘know’ or ‘understand’ the content they generate or if they are merely sophisticated pattern recognizers. The literature suggests a full spectrum of possibilities from the skeptical (e.g., Stechly et al., 2023; Ullman, 2023) to the sanguine (e.g., Kadavath et al., 2022; Kosinski, 2023). One marker of human-like reasoning is awareness and recognition of potential uncertainty or its corresponding confidence in the answer. Technically, LLMs use statistical prediction, but it is not obvious what confidence they implicitly and explicitly have in their responses. When we ask for an expert opinion, we usually expect it to come with some measure of confidence. This measure is standard in statistical expert systems, and a validated correlation between confidence level and reality plays a key role in establishing the systems’ credibility. Thus, our aim is to assess the degree of confidence LLMs have in their answers and how that confidence correlates with actual performance. The assessment of uncertainty in complex statistical problems is typically done using the bootstrap method. This will require access to raw data or some strong assumptions about the data distribution. Neither is feasible with the current LLMs, so we will instead rely on simple empirical methods. We measure confidence qualitatively and quantitatively as follows. For the former, the LLMs are prompted to reconsider their initial answers (regardless of their correctness). Presumably, an LLM is not going to change its mind if it is highly confident, and vice versa, it will change its mind if it has low confidence. For quantitative confidence, we ask them to explicitly tell us their confidence score in their responses. We also investigate the relationship between these confidence measures and the token-level probability produced by the LLM. We investigate three LLMs – GPT4o, GPT4-turbo and Mistral – and use two of the BIG Bench-Hard tasks (Suzgun et al., 2022): causal judgment (187 questions) and formal fallacies (250 questions). Furthermore, we assess the statistical reasoning abilities of LLMs in solving some probability and statistical puzzles and paradoxes (46 questions) from Pawitan and Lee (2024). To summarize briefly, in line with previous results, the LLMs perform significantly better than random guessing. However, when prompted to rethink their answers, they frequently change their mind and the overall accuracy of the second answers is often worse than that of the original answers, sometimes even worse than random guessing. Intriguingly, the tendency to change their mind depends on the phrasing of the prompt. There is a large discrepancy between qualitative and quantitative confidence, although we observe a significant correlation between them. When asked for confidence score, there is a strong tendency for overconfidence. The confidence measures are only partially explained by the underlying token-level probability. 2 Background 2.1 Testing the reasoning skills of an LLM Human intelligence is characterized not only by reasoning and understanding, but also introspection. Can LLMs, with their vast but opaque neural networks, claim similar capabilities? Their architectural complexity and the huge number of parameters ( \sim 175 billion for GPT3 and likely more than 1 trillion for the GPT4 series) have made their operations non-interpretable, much like the mysterious processes of our own brain. How do we assess novel reasoning abilities in machines? To be useful and informative, at the current stage of development we do not yet need to go to the ultimate Turing test (Turing, 1950). Traditional measures, such as the ability to recognize keywords, often just indicate a trained behavior, but do not necessarily reflect true cognitive skills. Tasks such as arithmetic calculations are too algorithmic and will offer little insight into emergent skills. (Even relatively recent LLMs, such as GPT3.5, are actually poor at arithmetic, but this issue is solved by the most recent ones, which can recognize and transfer the problem to specialized modules.) Many logical puzzles can be navigated through keyword recognition, making it difficult to discern truly novel reasoning. Emergent reasoning abilities, in contrast, would be indicated by an AI’s capacity to independently recognize and adapt to new problem patterns. What is needed are tests involving non-algorithmic and abstract reasoning challenges to better probe the depths of AI cognition. 2.2 Empirical studies The Beyond the Imitation Game Benchmark (BIG-Bench; Srivastava et al. 2022) is an extensive collaborative benchmark intended to probe LLMs in 204 cognitive and problem-solving tasks that are believed to be beyond the capabilities of LLMs. The tasks include linguistics, childhood development, mathematics, common-sense reasoning, biology, physics, social bias, software development, movie recommendation, etc. Indicating the level of interest and admirable commitment, the BIG-Bench was developed by 450 authors from 132 institutions. When the paper was first published in 2022, LLMs did not perform very well. For their normalized preferred metric, tasks are calibrated so that a score of 0 corresponds to poor performance and a score of 100 corresponds to very good performance. Human experts would be expected to achieve scores close to 100. When averaged on all tasks, the best performing language models achieved a score of less than 20. However, LLM performance has improved substantially; for instance, GPT4 performs similarly or better than the human in 17 of the 23 BIG-Bench Hard tasks (Zhou et al. 2024, Table 3). Another marker of reasoning is the ability to make plans. Valmeekam et al. (2023) tested some LLMs in the domains typically used in the International Planning Competition (IPC, 1998), including the well-known Blocks World, found that LLMs’ ability “to generate executable plans autonomously is rather limited, with the best model (GPT4) having an average