Why LLMs Cannot Think and How to Fix It

Marius Jahrens, Thomas Martinetz · 2025

Read Why LLMs Cannot Think and How to Fix It by Marius Jahrens, Thomas Martinetz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-09211.txt.

Introduction

Marius Jahrens and Thomas Martinetz argue that language models cannot think in a 2025 paper. They first define thought in a way that fits such models. The thought must take place inside the model's feature space. Classic views of thought inform their definition.

They then show current designs block that kind of thought. The block comes from both the architecture and the training method. Language modeling trains models to predict text, not to decide. The paper is a theory piece, not an experiment report. Fixes are proposed to let feature-space thought take place. The authors weigh what those changes would mean more broadly.

Catalog record

CREATORMarius Jahrens, Thomas Martinetz
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-09211

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Why LLMs Cannot Think and How to Fix It Marius Jahrens, Thomas Martinetz Abstract This paper elucidates that current state-of-the-art Large Language Models (LLMs) are fundamentally incapable of making decisions or developing "thoughts" within the feature space due to their architectural constraints. We establish a definition of "thought" that encompasses traditional understandings of that term and adapt it for application to LLMs. We demonstrate that the architectural design and language modeling training methodology of contemporary LLMs inherently preclude them from engaging in genuine thought processes. Our primary focus is on this theoretical realization rather than practical insights derived from experimental data. Finally, we propose solutions to enable thought processes within the feature space and discuss the broader implications of these architectural modifications. 1 Introduction Ever since language models have developed the ability to produce plausible texts and react to prompts and questions in a convincingly human style, there have been debates whether those text generations are merely plausible continuations based on statistical patterns or if there is some kind of internal planning taking place within the model such that the output is actually the lingual formulation of that internal information. Or in short: Do language models think? To address this question, we propose a definition for the concept of "thought" and demonstrate how it aligns with its traditional meaning in the context of human cognition. We then prove that current LLMs, by design, are incapable of forming any such thought. We begin by exploring the theoretical foundations of thought and its relevance to LLMs. Following this, we examine the consequences of the current architectural limitations, focusing on how they negatively impact the efficiency and capabilities of LLMs. To address these shortcomings, we identify two critical design decisions responsible for the lack of thought processes in LLMs and propose several practical solutions to alleviate these issues. Our primary contribution is the theoretical realization that LLMs, as they are currently designed and trained, cannot engage in genuine thought processes. Additionally, we offer practical insights and architectural modifications aimed at enabling thought processes within the feature-space, thereby enhancing the reasoning and planning capabilities of LLMs. 2 Related work 2.1 Reasoning in text space Various attempts have been made to enhance planning and reasoning capabilities in LLMs by making them to formulate lines of reasoning or step-by-step operations within the text space. Notably, Chain-of-Thought by Wei et al. 2022 and Tree-of-Thought by Yao et al. 2023 have pursued this approach, with Madaan et al. 2023 further enhancing it through a self-refinement loop. More recently, Quiet-STaR by Zelikman et al. 2024 has combined formulated lines of reasoning with ground-truth solutions to fine-tune models for more reliable and accurate verbalized reasoning outputs. However, these efforts still confine reasoning processes to the text space. Studies by Huang et al. 2024 , Tyen et al. 2024 , and Turpin et al. 2023 have highlighted the limitations of text-space reasoning. As we will demonstrate in this work, choices made by the model in these scenarios are either the result of the random sampling process at the model’s output, preconditioned by training data preferences, or predetermined by the conversation history. These choices are never made within the feature space of the model. 2.2 Reasoning in feature space In a recent study, Goyal et al. 2024 explored training an LLM for problem-solving in a chat format by inserting pause tokens into the conversation prior to decoding the model’s response. They incentivised the model to use the additional feature tensors to generate relevant features for problem-solving responses, thereby investigating the impact of providing additional space in the model state on reasoning task performance. Another approach by Pfau et al. 2024 examined the feasibility of replacing a chain-of-thought with meaningless filler tokens to encourage the model to perform reasoning in the feature space in its place. Additionally, while not directly focused on reasoning, Wu et al. 2024 investigated how a transformer model’s internal features influence future outputs. However, their results do not fully align with our stricter definition of "thinking ahead" as used in this work. In summary, while significant progress has been made in reasoning within the text space, the exploration of reasoning within the feature space remains limited. Our work aims to bridge this gap by enabling and evaluating genuine thought processes within the feature space of LLMs. 3 Defining "thought" 3.1 A thought experiment To informally introduce the concept of "thought", consider the following thought experiment involving an LLM chatbot: User: Let’s play Rock Paper Scissors. Think of the one you are going to pick, but don’t tell me yet. Let me know when you have made your choice. Assistant: Got it, I have chosen one. User: Alright, what did you choose? Assistant: (processing...) At this point in the conversation, the question is if the LLM has actually made a choice or not. This choice would not yet be written down, so it exists purely within the model’s internal state. If the model has made a choice, we would expect the output probability distribution for the three options (rock, paper, scissors) to collapse towards one of the options. Conversely, if the probability distribution has not collapsed towards one of the options, then we say the model has not made a choice. This kind of hidden information, that influences the output probabilities and is not a deterministic function of the observable input, is what we refer to as a "thought". The non-determinism criterion distinguishes thoughts from reflexes, muscle memory, personal bias, or character traits, which are all more hardwired and less transient. Note, that even if we were to include deterministic thoughts in this definition, the issues presented in chapter 4 would still prevent the model from collapsing the output probability distribution towards one of the options. 3.2 Mathematical formulation Statistically, the influence of hidden information contained within the model state can be measured using conditional mutual information, as given in equation 1 . Here, X_{1},\dots,X_{t} represent the input tokens up until the t -th token in the sequence, Y_{t} represents the model output, and State_{t} represents the model state. I(State_{t};Y_{t}|X_{1,\dots,t}) (1) 4 Why current LLMs cannot think The inability of current Large Language Models to develop thoughts or make decisions in feature space arises from two key design choices. 4.1 Deterministic model state Current state-of-the-art LLMs are based on decoder-only transformer architectures. In these models, the input sequence consists of prompts in the form of texts and images, as well as the model’s sampled outputs. Consequently, the model’s state, represented by its internal feature tensors, is entirely deterministically determined by the input sequence, i.e. the user inputs and the model’s responses. Mathematically, this can be expressed as: State_{t}=f(X_{1},...,X_{t}) (2) The only source of randomness in these models is the sampling process from the model’s output token probability distribution. Given this deterministic nature, the entropy of the model output given the input sequence remains unchanged, irrespective of whether the model state is known: H(Y_{t}|X_{1,\dots,t})-H(Y_{t}|State_{t},X_{1,\dots,t})=0. (3) The expression above is the conditional mutual information between the model output and the model’s state and it follows immediately that: I(State_{t};Y_{t}|X_{1,\dots,t})=0 (4) Therefore, model decisions cannot take place in feature-space - i.e. materialize as a collapse of the output distribution - unless the input sequence

Read the complete text