Can LLMs subtract numbers?

Mayank Jobanputra, Nils Philipp Walter, Maitrey Mehta, Blerta Veseli, Evan Parker Kelly Chapple, Yifan Wang, Sneha Chetani, Ellie Pavlick, Antonio Vergari, Vera Demberg, [1ex] Saarland University · 2025

Read Can LLMs subtract numbers? by Mayank Jobanputra, Nils Philipp Walter, Maitrey Mehta, Blerta Veseli, Evan Parker Kelly Chapple, Yifan Wang, Sneha Chetani, Ellie Pavlick, Antonio Vergari, Vera Demberg, [1ex] Saarland University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-02795.txt.

Introduction

Mayank Jobanputra and nine colleagues test a basic skill in large language models: subtraction. Addition and multiplication get most of the attention in math benchmarks, but subtraction differs because order matters and answers can go negative. Their 2025 study runs eight pretrained models from four families on both operations and compares the scores.

Subtraction accuracy lagged addition by a wide margin. Errors clustered where the second number was larger, so the true answer was negative. There the models often gave the right size but dropped the minus sign. Probing showed the models did encode the sign inside, yet failed to print it. Few-shot examples helped a little. Instruction tuning nearly fixed the sign problem, suggesting the weakness is real but recoverable.

Catalog record

CREATORMayank Jobanputra, Nils Philipp Walter, Maitrey Mehta, Blerta Veseli, Evan Parker Kelly Chapple, Yifan Wang, Sneha Chetani, Ellie Pavlick, Antonio Vergari, Vera Demberg, [1ex] Saarland University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-02795

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Can LLMs subtract numbers? Mayank Jobanputra, Nils Philipp Walter, Maitrey Mehta, Blerta Veseli, Evan Parker Kelly Chapple, Yifan Wang, Sneha Chetani, Ellie Pavlick, Antonio Vergari, Vera Demberg, [1ex] Saarland University Abstract We present a systematic study of subtraction in large language models (LLMs). While prior benchmarks emphasize addition and multiplication, subtraction has received comparatively little attention despite being structurally distinct as a non-commutative operation. We evaluate eight pretrained LLMs spanning four families on addition and subtraction problems. Our experiments reveal that subtraction accuracy lags behind addition by a wide margin. We find that the errors for ( a − b a-b ) are concentrated in cases where ( a < b a<b ). In such cases, LLMs frequently produce the correct magnitude but omit the negative sign. Probing analyses show that LLMs internally encode whether results should be negative, yet this information is often not reflected in generated outputs. We further test well-known techniques such as few-shot learning and instruction-tuning to see if they can improve the LLMs’ performance. Our results suggest that while few-shot prompting yields modest gains, the instruction-tuned models achieve near-perfect accuracies in generating the negative sign. Together, these findings provide a clearer characterization of the limitations and recoverability of LLMs’ arithmetic capabilities in subtraction. 1 Introduction Large language models (LLMs) show strong performance on arithmetic tasks, particularly when aided by chain-of-thought (CoT) prompting and its variants Wei et al. 2022 ; Kojima et al. 2022 ; Imani et al. 2023 ; Li et al. 2025 . However, prior work and widely used benchmarks Balunović et al. 2025 ; Yang et al. 2025b ; Hendrycks et al. 2021 concentrate on addition and multiplication, leaving subtraction comparatively underexplored. This gap raises the question of how well LLMs handle subtraction, especially in a zero-shot setting. Subtraction differs from addition in an important structural way as it is a non-commutative operation ( a-b\neq b-a ). This makes the order of operands decisive for subtraction. Moreover, subtraction accentuates better positional representations to facilitate accurate borrowing. Borrowing requires robust tracking of digit positions across potentially long sequences — a setting where transformers trained from scratch are known to struggle McLeish et al. 2024 . These factors pose additional demands unique to subtraction, making it unclear whether LLMs can sustain their strong performance on addition Zhou et al. 2024 ; Nikankin et al. 2025 when applied to subtraction. In this work, we systematically evaluate subtraction across LLM families and sizes, benchmarking against addition to establish a clear performance baseline. We evaluate eight pretrained LLMs of different sizes from four model families, Gemma-2 Riviere et al. 2024 , Qwen3 Yang et al. 2025a , OLMo-2 OLMo et al. 2024 , and Llama-3 Grattafiori et al. 2024 . We find a surprising disparity between the additive and subtractive capabilities of LLMs. Our findings reveal that pretrained LLMs struggle to generate correct answers when subtraction results are negative. Our main findings are: a) Under matched complexity, subtraction performance consistently lags addition in state-of-the-art open-source LLMs. In some cases, LLMs reach perfect accuracy on addition while only about half that level on subtraction. (§ 3.1 ) b) Subtraction a-b is disproportionately error-prone when (a<b) , with a dominant failure mode in which LLMs produce the correct magnitude but omit the negative sign. (§ 3.2 , 3.3 ) c) Our probing results suggest that LLMs internally encode when the result should be negative, but they often fail to surface this at decoding time. (§ 3.4 ) d) Instruction tuning improves subtraction performance to levels comparable with addition in a zero-shot setting. However, adding a few-shot examples yields modest and inconsistent improvements across pretrained LLMs. (§ 4 ) This candidates subtraction as an inherently harder task than addition, and can help shed light on the different regimes (i.e. pretrained or instruction-tuned) in which LLMs learn and reason. 2 Experimental Setup Following prior work Zhou et al. 2024 ; Nikankin et al. 2025 , we primarily focus on single-token numbers for our analysis. However, we also analyze multi-token numbers in Appendix A that show similar results. 2.1 Data generation To evaluate LLMs on the integer subtraction task, we generate synthetic datasets using controlled settings described below. Sampling operands a and b . We sample operands a and b uniformly from the range of numeric values that are represented as single tokens by each LLM’s tokenizer (see Tokenizer Range in Table 1 ). Consequently, each LLM is evaluated only on problems involving numbers within its own tokenizer range. For every sampled pair (a,b) , we generate arithmetic problems and categorize them based on the relationship between operands (e.g., a>b , a<b , a=b ). The final dataset (see #Used Samples in Table 1 ) is balanced, containing an equal number of problems with a>b and a<b . For our work, we do not make use of operands where a=b . LLM Family Tokenizer Range Total Samples #Samples a>b #Samples a<b Prompt Variants #Samples Used Qwen3 [0, 9] 100 45 45 5 450 Gemma-2 [0, 9] 100 45 45 5 450 Llama-3 [0, 999] 1,000,000 499,500 499,500 5 10,000 OLMo-2 [0, 999] 1,000,000 499,500 499,500 5 10,000 Table 1: Dataset statistics for subtraction experiments across different LLM families. Tokenizer Range indicates the continuous range of numeric values that are single tokens for each LLM. Total Samples reflects the total number of possible subtraction problems. For LLMs where Total Samples exceeds 2000, we uniformly sample 2000 problems from the full set. We then apply our prompt variants to each sampled problem and obtain all the samples for inference (i.e. #Samples Used ). Prompt Variants. To ensure the generalizability of our results and minimize the risk of spurious correlations, we use five different prompt formats, ranging from minimal equation style input to verbose template style input. We provide the exact prompt templates in the Appendix D . Zero-shot vs. n-shot. Zero-shot prompts contained only the query equation formatted under one of the five variants. N-shot prompts include solved examples (up to ten) before the query, allowing us to probe in-context learning abilities of LLMs. We use the same procedure to generate the data for both addition ( + ) and subtraction ( - ) operators. This procedure yields a balanced dataset, and systematically varied across operators, operand order, and prompt phrasings. We provide full dataset statistics in Table 1 . 2.2 LLM Inference For pretrained LLMs, we use greedy sampling with a temperature of 0 and sample up to 20 new tokens. For instruction-tuned LLMs, we use the sampling strategy and parameters suggested by the LLM creators. Additionally, we use the default system prompt for each instruction-tuned LLM and sample up to 500 new tokens. We use a fixed seed across all experiments to ensure reproducibility. We extract the final numerical answer from the LLMs’ generated text using a robust parsing mechanism. We run all experiments on 4x H100 GPUs using vLLM Kwon et al. 2023 without any quantization. 3 Results In this section, we analyze LLM performance on subtraction, using addition as a reference point. We examine how accuracy varies across LLMs, operand order, and analyze the errors that emerge. 3.1 Can LLMs subtract numbers? We start our exploration by comparing subtraction accuracy

Read the complete text