Multiplication in Multimodal LLMs: Computation with Text, Image, and Audio Inputs

Samuel G. Balter, Ethan Jerzak, Connor T. Jerzak · 2026

Read Multiplication in Multimodal LLMs: Computation with Text, Image, and Audio Inputs by Samuel G. Balter, Ethan Jerzak, Connor T. Jerzak (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18203.txt.

Introduction

Balter, Ethan Jerzak, and Connor Jerzak test whether multimodal language models can actually multiply across text, image, and audio inputs. Models perceive numbers well in every modality yet fail at exact multi-digit multiplication. The authors build a paired benchmark varying digit length, sparsity, representation, and modality, and define arithmetic load C from total and nonzero digit counts. Accuracy falls sharply as C grows, nearing zero past 100, while perception checks stay near-perfect, showing the failure is computational, not perceptual.

Catalog record

CREATORSamuel G. Balter, Ethan Jerzak, Connor T. Jerzak
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55506
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2604.18203

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multiplication in Multimodal LLMs: Computation with Text, Image, and Audio Inputs Thanks: To appear in ACL Findings (2026). Samuel G. Balter, Ethan Jerzak, Connor T. Jerzak Abstract Multimodal LLMs can accurately perceive numerical content across modalities yet fail to perform exact multi-digit multiplication when the identical underlying arithmetic problem is presented as numerals, number words, images, or in audio form. Because existing benchmarks often lack systematically paired instances across modalities, it remains difficult to compare genuine arithmetic limits within and across model families. We therefore introduce a controlled multimodal multiplication benchmark that factorially varies digit length, digit sparsity, representation (e.g., numerals vs. number words), and modality (text, rendered images, and audio), with paired instances from a reproducible generator. We also define arithmetic load, C C , as the product of the Total and Non-Zero digit count as a compact, mechanistically motivated proxy for operation count. Across evaluations, accuracy falls sharply as C C grows, often nearing zero by C > C> 100. Indeed, C C remains predictive of performance across modalities and models, with R 2 R^{2} > > 0.5, nearing the value from more complex measures of arithmetic load that count the number of intermediate arithmetic steps. A separate perception-versus-computation decomposition shows that multimodal degradation is primarily computational rather than perceptual: on matched perception checks, models are near-perfect ( > > 99%) across modalities even when multiplication accuracy drops substantially. Beyond measuring when models fail, we ask which procedures they are predisposed to follow. We introduce a style-controlled forced-completion loss probe that scores heuristic-specific reasoning prefixes—including columnar multiplication, distributive decomposition, and rounding/compensation. Here, distributive decomposition is favored in both text and vision modalities; heuristic-specific LoRA adapters produce near-orthogonal updates yet degrade accuracy, indicating the base model maintains a well-tuned internal router. 1 Introduction Large language models (LLMs) increasingly act as general-purpose assistants, and many real workflows require exact arithmetic. Unfortunately, arithmetic competence can be unpredictable: a model that correctly computes 47 \times 36 may fail on a nearby instance like 89 \times 67, despite both being two-digit multiplications. Recent evaluations confirm that multi-digit arithmetic can be brittle in LLMs ( Mahendra et al. 2025 ) ; performance can vary with operand size and surface form in ways that are hard to anticipate ( Yuan et al. 2023 ) . Methods such as chain-of-thought prompting and math-focused pretraining improve performance on broader quantitative reasoning tasks ( Wei et al. 2022 ; Lewkowycz et al. 2022 ) , but they do not guarantee the reliable execution of the intermediate calculations those approaches rely on. Moreover, widely used benchmarks (e.g., GSM8K) can entangle language understanding, multi-step reasoning, and computation, making it difficult to attribute errors to arithmetic rather than to interpretation or planning ( Cobbe et al. 2021 ) . At the same time, foundation models are increasingly multimodal , with arithmetic questions often arriving as rendered equations in images (e.g., screenshots, slides, scanned homework) or audio traces instead of as text tokens ( Alayrac et al. 2022 ; OpenAI 2023 ) . This issue becomes more important in agentic workflows: an agent may inspect a screenshot, choose a calculation path, call tools, and verify intermediate states. These steps, however, are most effective when the model captures the mathematical problem underlying the model presentation completely, instead of relying on superficial cues in answering. Recent benchmarks and training efforts in multimodal mathematical reasoning emphasize complex visual contexts and diverse problem types ( Lu et al. 2023 ; Shi et al. 2024 ) , but comparatively little work isolates elementary arithmetic under controlled problem structure and controlled modal format. This gap matters because representation and perception can amplify small computational weaknesses: the same multiplication might be easy as typed numerals but hard when embedded in a noisy visual channel or reformulated as spoken number words. We address this gap with controlled experiments on multiplication, a particularly sensitive probe because it requires carry propagation and long-range digit interactions. In the benchmark, we systematically vary (i) operand digit length, (ii) the number of non-zero digits, and (iii) modality (paired text, rendered images, and audio). Through this benchmark, we ask (1) How do digit length and algorithmic load (total digits \times non-zero digit count) determine accuracy? (2) How much does performance change when the same computation is presented across modalities and representations? (3) Which internal arithmetic heuristics do models prefer, how do these priors shift with operand cues and modality, and do they help explain failure patterns? Beyond accuracy, we will aim to characterize why failures occur by probing strategy preference. Human multiplication draws on multiple heuristics—including columnar procedures, distributive decomposition, and rounding/compensation—whose use depends on problem structure ( Campbell 2005 ) . Inspired by probing methods for interpreting neural representations ( Belinkov 2022 ) , we introduce a forced-completion loss probe over heuristic-specific reasoning preambles, using only forward passes. Finally, using low-rank adapters ( Hu et al. 2022 ) , we analyze whether heuristic-specialized updates align in parameter space, providing geometric evidence about whether these heuristics correspond to distinct mechanisms. Our contributions are: • Controlled multimodal benchmark. We introduce a reproducible multiplication benchmark that factorially varies digit length, digit sparsity, representation, and modality under standardized evaluation across modalities. • An interpretable difficulty measure. We show that arithmetic load (total digits \times non-zero digits) is a strong predictor of accuracy and helps summarize degradation across modalities. • Forced-completion loss-based strategy fingerprinting. We use a forward-pass loss probe and find model-specific low-loss preferences, with margins shifting with operand cues and modality. • Geometric evidence for distinct heuristics. Heuristic-specific LoRA adapters show separated effective update directions (cosine similarity of \Delta W=BA ), with stronger orthogonality in 235B than in 30B, suggesting distinct parameter subspaces. 47\times 36 (same problem) “What is 47\times 36 ?” “forty-seven times…” 47\!\times\!36\!=\!? audio Multimodal Inputs MM-LLM e.g., Qwen3-VL 30B / 235B Arithmetic Load C=d_{\text{total}}\!\times\!d_{\text{non-zero}} 4\times 4=16 Processor Forced-Completion Loss Probes (Cross-Entropy) OT Columnar 7.09/7.25 DD Decomposition 7.02/6.38 ✓ RC Rounding 7.08/6.78 \leftarrow lower indicates deeper alignment Cross-Modal Text \approx Image preferences DD preferred in both \approx LoRA Orthogonality OT \perp DD \perp RC \rho\approx 0 (distinct subspaces) Key: DD preferred (text/image lowest-loss match, 30B: 68%/91%; 235B: 93%/100%) \mid OT rarely lowest-loss match (text/image, 30B: 24%/16%; 235B: 0%/0%) \mid Heuristics orthogonal ( \rho\approx 0 ) Figure 1: Overview of our arithmetic benchmark and heuristic fingerprinting methodology for multimodal LLMs. In the generator, the same multiplication problem (e.g., 47\times 36 ) is paired across numeric text, rendered numeric image, and audio. 2 Background 2.1 Arithmetic Reliability in LLMs To situate our controlled benchmark, we briefly summarize what prior work has established about exact arithmetic in LLMs. Although LLMs can produce fluent arithmetic explanations, controlled evaluations show that exact multi-digit computation can be brittle: accuracy drops quickly as digit length grows and carry interactions accumulate, while small changes in operand configuration can flip correctness even under deterministic decoding ( Yuan et al. 2023 ) . Prompting methods such as chain-of-thought can improve performance on many reasoning benchmarks by eliciting intermediate steps ( Wei et al. 2022 ) , and math-focused pretraining and verification-style training can further raise end-to-end scores on quantitative tasks ( Lewkowycz et al. 2022 ; Cobbe et al. 2021 ) . However, these approaches still

Read the complete text