Even GPT-5.2 Can’t Count to Five: The Case for Zero-Error Horizons in Trustworthy LLMs
Ryoma Sato [email protected] · 2026
Read Even GPT-5.2 Can’t Count to Five: The Case for Zero-Error Horizons in Trustworthy LLMs by Ryoma Sato [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-15714.txt.
Introduction
Ryoma Sato proposes a strict yardstick for language models in a 2026 arXiv paper. He calls it the Zero-Error Horizon. It is the maximum range over which a model solves tasks with no errors at all. Averages can hide rare slips. The horizon exposes them: one slip ends the run. Sato argues that this simple measure yields rich insight. He tests it on state-of-the-art models.
His example is pointed. Even GPT-5.2, he reports, fails at counting to five under his test. The title uses that failure as a warning. Trustworthy use needs error-free stretches, not just high average scores. A surgeon-grade tool must be right every time in its working range. The horizon frames exactly how far that range extends for each model.
Catalog record
| CREATOR | Ryoma Sato [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-15714
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Even GPT-5.2 Can’t Count to Five: The Case for Zero-Error Horizons in Trustworthy LLMs Ryoma Sato [email protected] Abstract We propose Zero-Error Horizon (ZEH) for trustworthy LLMs, which represents the maximum range that a model can solve without any errors. While ZEH itself is simple, we demonstrate that evaluating the ZEH of state-of-the-art LLMs yields abundant insights. For example, by evaluating the ZEH of GPT-5.2, we found that GPT-5.2 cannot even compute the parity of a short string like 11000, and GPT-5.2 cannot determine whether the parentheses in ((((()))))) are balanced. This is surprising given the excellent capabilities of GPT-5.2. The fact that LLMs make mistakes on such simple problems serves as an important lesson when applying LLMs to safety-critical domains. By applying ZEH to Qwen2.5 and conducting detailed analysis, we found that while ZEH correlates with accuracy, the detailed behaviors differ, and ZEH provides clues about the emergence of algorithmic capabilities. Finally, while computing ZEH incurs significant computational cost, we discuss how to mitigate this cost by achieving up to one order of magnitude speedup using tree structures and online softmax. 1 Introduction Large Language Models (LLMs) are being applied to an increasingly wide range of domains. In particular, their use in safety-critical domains such as healthcare, law, finance, and science is being actively explored [ 1 , 11 , 34 , 54 , 66 , 64 ] . While LLMs appear extremely intelligent and capable of reasoning, they sometimes make mistakes that seem inconceivably foolish from a human perspective. For example, GPT-5.2 can implement complex fluid dynamics simulation code, yet it cannot even compute the parity of the short string 11000 , cannot determine whether the parentheses in ((((()))))) are balanced, and makes calculation errors on 127\times 82 (Figure 1 ). This inconsistency poses a major obstacle when applying LLMs to safety-critical domains. What if an AI conducting large-scale financial transactions employs sophisticated financial theories but then makes a calculation error on 127\times 82 and incurs massive losses? What if an AI managing a nuclear reactor system mistakenly believes that the status flag 11000 contains an odd number of 1s and opens the door of an operating reactor? In addition to the API commands in Figure 1 , GPT-5.2 appears to be particularly poor at handling the parenthesis string ((((()))))) . As of January 22, 2026, when queried through the web interface, even with chain-of-thought reasoning (GPT-5.2-Thinking), it often fails to correctly determine whether ((((()))))) is balanced (Figure 2 ). While the web interface behavior is stochastic, after trying many patterns, it frequently answered that ((((()))))) was balanced. We encourage readers to try this. In this paper, we propose Zero-Error Horizon (ZEH) as a tool for clarifying such “holes” in AI capabilities. A model having \text{ZEH}=n for a given problem means that it can solve all instances up to size n without error, but makes at least one error on instances of size n+1 . For example, in computing the parity of binary strings, if we define problem size as string length, GPT-5.2 can accurately compute parity for strings up to 4 characters, but makes an error on 11000 , so \text{ZEH}=4 . For multiplication a\times b , if we define problem size as the larger operand value \max(a,b) , GPT-5.2 correctly answers all multiplications where both operands are at most 126, but makes an error on 127\times 82 , so \text{ZEH}=126 . The definition of ZEH itself is not novel. However, we discovered that applying ZEH to LLMs automatically yields many surprising results. There are numerous ways to use ZEH. First, ZEH provides a guideline for the problem sizes that can be safely entrusted to an LLM. Since \text{ZEH}=126 , we can say that GPT-5.2 can be trusted to some extent (though not with certainty, as we explain later) for mental arithmetic of two-digit multiplications. Conversely, one should be cautious about entrusting GPT-5.2 with tasks involving three-digit multiplications. While this might be acceptable for everyday shopping, in mission-critical tasks, the appearance of three-digit multiplication could serve as a warning signal worth considering. Similarly, with \text{ZEH}=10 for balanced parentheses, GPT-5.2 might be trusted with nested structures of about 5 pairs, but one should be more cautious beyond that. Figure 1 : GPT-5.2 cannot compute the parity of the short string 11000 , cannot determine whether the parentheses in ((((()))))) are balanced, and makes a calculation error on 127\times 82 . These commands can be copied and pasted, and if you set your API key $OPENAI_API_KEY , you can try them right now. Please give it a try. Figure 2 : GPT-5.2-Thinking also fails to determine whether the parentheses in ((((()))))) are balanced. Additionally, we confirm that analyzing ZEH provides insights into the reasoning modes of LLMs. Our main claim in this paper is that ZEH has numerous properties that accuracy lacks, and measuring ZEH is useful when investigating LLM performance. In particular, measuring ZEH is important when applying LLMs to safety-critical domains. Code is available at https://github.com/joisino/zeh . 2 Zero-Error Horizon 2.1 Definition We fix a model, problem, and prompt. For example, the model is GPT-5.2-2025-12-11 , the problem is multiplication, and the prompt is {"instructions": "Answer with only the integer.", "input": "{a}*{b}="} . Let B_{n} denote the set of problems of size n , and let T_{n}=\bigcup_{i=1}^{n}B_{i} denote the set of problems of size at most n . Example 1 (Multiplication). • B_{n}=\{(a,b)\mid\max(a,b)=n\} • T_{n}=\{(a,b)\mid a\leq n,b\leq n\} Example 2 (Parity of binary strings). • B_{n}=\{0,1\}^{n} • T_{n}=\{\text{all binary strings of length}\leq n\} Example 3 (Balanced parentheses). • B_{n}=\{"(",")"\}^{n} • T_{n}=\{\text{all strings of "(" and ")" of length}\leq n\} Example 4 (Chromatic number). • B_{n}=\{\text{graphs with }n\text{ nodes}\} • T_{n}=\{\text{graphs with at most }n\text{ nodes}\} Let C denote the set of problems that the fixed model, problem, and prompt answer correctly under deterministic greedy decoding. We define the Zero-Error Horizon (ZEH) as follows: \text{ZEH}=\max\{n\mid T_{n}\subseteq C\} (1) That is, the model can solve all instances up to size ZEH without error, but makes at least one error on instances of size \text{ZEH}+1 . We call an instance of size \text{ZEH}+1 on which the model makes an error a ZEH limiter . 2.2 Qualitative Characteristics of ZEH 2.2.1 Safety Guarantees and Warning Signals ZEH can be used for both safety arguments—“this size can be safely entrusted”—and danger arguments—“beyond this point is dangerous.” Unlike metrics such as accuracy that evaluate performance within a predetermined range, ZEH’s characteristic of evaluating the range itself works effectively. By determining the boundary, we can argue that inside the boundary is safe and outside is dangerous. In safety-critical operations, even if average performance is high, holes in capability like careless mistakes can be fatal. Sampling problems may miss such holes, which is problematic in safety-critical domains. ZEH ensures no oversights through exhaustive verification. It is important to note that problems within ZEH are not necessarily absolutely safe for the model. The definition of ZEH fixes the prompt (and context). For example, Qwen2.5-3B-Instruct has a ZEH of 15 on the multiplication task when using the prompt • Baseline: {"instructions": "Answer with only the integer.", "input": "{a}*{b}="} but may make errors on size 15 problems with other prompts or contexts. In fact, with the prompt • compute: