LLM Knows Geometry Better than Algebra: Numerical Understanding of LLM-Based Agents in A Trading Arena
Tianmi Ma, Jiawei Du, Wenxin Huang, Wenjie Wang, Liang Xie, Xian Zhong, and Joey Tianyi Zhou · 2025
Read LLM Knows Geometry Better than Algebra: Numerical Understanding of LLM-Based Agents in A Trading Arena by Tianmi Ma, Jiawei Du, Wenxin Huang, Wenjie Wang, Liang Xie, Xian Zhong, and Joey Tianyi Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-17967.txt.
Introduction
Chat models read charts better than raw numbers in a virtual stock market, Tianmi Ma and co-authors find in a 2025 paper. Language model traders compete in a zero-sum arena where their own bids move prices, unlike backtests on frozen history. Bid and ask orders meet as in a live exchange.
Given plain text numbers, the agents reason poorly and chase recent values. Swap the same data into charts and both number sense and profit jump. A reflection step adds further gains. The arena gives a fair test bed for machine investors and exposes a plain truth: format shapes machine judgment.
Catalog record
| CREATOR | Tianmi Ma, Jiawei Du, Wenxin Huang, Wenjie Wang, Liang Xie, Xian Zhong, and Joey Tianyi Zhou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-17967
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLM Knows Geometry Better than Algebra: Numerical Understanding of LLM-Based Agents in A Trading Arena Tianmi Ma, Jiawei Du, Wenxin Huang, Wenjie Wang, Liang Xie, Xian Zhong, and Joey Tianyi Zhou Abstract Recent advancements in large language models (LLMs) have significantly improved performance in natural language processing tasks. However, their ability to generalize to dynamic, unseen tasks, particularly in numerical reasoning, remains a challenge. Existing benchmarks mainly evaluate LLMs on problems with predefined optimal solutions, which may not align with real-world scenarios where clear answers are absent. To bridge this gap, we design the Agent Trading Arena , a virtual numerical game simulating complex economic systems through zero-sum games, where agents invest in stock portfolios. Our experiments reveal that LLMs, including GPT-4o, struggle with algebraic reasoning when dealing with plain-text stock data, often focusing on local details rather than global trends. In contrast, LLMs perform significantly better with geometric reasoning when presented with visual data, such as scatter plots or K-line charts, suggesting that visual representations enhance numerical reasoning. This capability is further improved by incorporating the reflection module, which aids in the analysis and interpretation of complex data. We validate our findings on NASDAQ Stock dataset, where LLMs demonstrate stronger reasoning with visual data compared to text. Our code and data are publicly available at https://github.com/wekjsdvnm/Agent-Trading-Arena.git . † † footnotetext: Email:[email protected], [email protected]. ∗ denotes the equal contribution. ✉ represents the corresponding author. 1 Introduction Recent advancements in large language models (LLMs) have demonstrated exceptional proficiency across various domains, achieving state-of-the-art performance in natural language processing (NLP) tasks such as translation ( Koshkin et al. 2024 ) , summarization ( Yang et al. 2023 ) , and reasoning ( Kalyanpur et al. 2024 ; Qiao et al. 2023 ) . While LLMs excel in language-based tasks, further progress in numerical and geometric reasoning is essential to tackling complex, interdisciplinary challenges, particularly in fields like finance and scientific research. Benchmarks such as GSM8K ( Cobbe et al. 2021 ) and MATH ( Hendrycks et al. 2021 ) have been developed to assess and improve the mathematical problem-solving abilities of LLMs through structured datasets and standardized evaluation protocols. These benchmarks not only advance LLM development but also represent a crucial step in enhancing the ability of artificial intelligence systems to solve mathematical reasoning problems. Figure 1: Illustration and Performance Comparison of Textual and Visual (K-Line Chart) Numerical Inputs. Top: The left figure presents numerical data in K-line chart format, while the right figure shows textual numerical inputs. Bottom: LLMs with visual inputs (orange) significantly outperform those with textual inputs (blue) in 40-day yield trends, demonstrating the advantage of visual data in financial decision-making. In response, recent updates to LLMs have focused on improving their mathematical abilities, with models ( Ahn et al. 2024 ; Romera-Paredes et al. 2024 ) achieving new records on math-focused benchmarks. However, many of these benchmarks ( Taylor et al. 2022 ; Naveed et al. 2023 ) primarily evaluate performance on problems familiar to the models, closely resembling those encountered during training and often following standard formats, solution strategies, or recurring mathematical patterns. This raises concerns about whether LLMs genuinely possess reasoning abilities for novel numerical data or whether their success is largely due to memorization and pattern recognition. These limitations underscore the need for alternative evaluation paradigms that assess LLMs’ ability to generalize mathematical principles to unseen scenarios. To address these limitations, we introduce the Agent Trading Arena , a virtual numerical game generating diverse numerical data through agent interactions in a zero-sum stock market environment. In this system, LLM-based agents make trading decisions based on historical stock prices, dynamically determined by agents’ bidding activities. Unlike static benchmarks ( Naveed et al. 2023 ) with predefined strategies, our simulation forces agents to adapt to evolving market conditions. Any temporarily optimal strategy is quickly countered, fostering an environment that rewards adaptability. By mimicking competitive trading scenarios where outcomes depend on real-time agent decisions, we create a controlled yet dynamic environment that tests LLMs’ ability to infer hidden patterns and generalize numerical laws without relying on memorization or prior knowledge. Our experiments reveal that LLMs struggle with textual numerical data, often focusing on absolute values rather than capturing percentage changes and relations between data points, akin to algebraic reasoning. Additionally, LLMs tend to overemphasize recent data while overlooking earlier information, even when explicitly highlighted in the prompt. These limitations suggest that LLMs lack the ability to abstract numerical information into higher-level representations, a crucial skill for generalizing beyond explicit values. In contrast, LLMs perform better when presented with numerical data in visualized formats, such as scatter plots, line charts, and bar graphs, which involve geometric reasoning. Our experiments show that LLMs processing visual numerical data consistently outperform those processing textual numerical data. As shown in Figure 1 , visual numerical data enables LLMs to achieve significantly higher return rates in the Agent Trading Arena , underscoring the advantages of structured visual representations. Incorporating the reflection module ( Zhou et al. 2022 ; Shinn et al. 2023 ) further emphasizes the performance gap between textual and visual data, with the module proving particularly effective for visual representations. By leveraging structured visualizations, the reflection module enhances the model’s ability to reason through complex data relations, leading to more accurate and strategic decision-making. The Agent Trading Arena provides a valuable framework for advancing LLM generalization and adaptability, with broad applications in fields like finance, healthcare, and scientific research. To further validate our findings, we extend our evaluation to real-world financial data, such as NASDAQ Stock dataset, which includes historical trading data. The consistent results suggest an important contribution: LLMs exhibit stronger reasoning with visual geometric data compared to textual numerical data, demonstrating the value of visual representations in enhancing LLM performance. 2 Related Works 2.1 Mathematical Benchmarks for LLMs Math word problems (MWPs) have been widely studied, leading to the development of various benchmarks for evaluating models’ mathematical reasoning and problem-solving abilities. Early datasets, such as MAWPS ( Koncel-Kedziorski et al. 2016 ) , standardized existing problems to facilitate consistent evaluation. Math23K ( Wang et al. 2017 ) introduced a large-scale collection of Chinese arithmetic problems that require structured equation solving. To increase diversity, benchmarks like ASDiv ( Miao et al. 2020 ) and SVAMP ( Patel et al. 2021 ) provide richer annotations and a broader range of problem types. More recent benchmarks, including GSM8K ( Cobbe et al. 2021 ) and MATH ( Hendrycks et al. 2021 ) , focus on multi-step reasoning and advanced mathematical concepts, broadening the scope of evaluation. Additionally, MathQA-Python ( Austin et al. 2021 ) , a Python variant of MathQA ( Amini et al. 2019 ) , emphasizes programmatic reasoning, while MGSM ( Shi et al. 2023 ) extends these benchmarks to multilingual contexts. Despite these advancements, current models ( Lu et al. 2023b ; Lu et al. 2023a ) primarily rely on memory-based answering strategies learned through extensive training, rather than demonstrating true mathematical reasoning. 2.2 LLMs for Enhanced Mathematical Reasoning Building on these benchmarks, LLMs have advanced mathematical problem-solving by incorporating specialized datasets into their training. Models such as Galactica ( Taylor et al. 2022 ) , PaLM-2 ( Anil et al.