Can Language Models Rival Mathematics Students? Evaluating Mathematical Reasoning through Textual Manipulation and Human Experiments

Andrii Nikolaiev, Yiannos Stathopoulos, Simone Teufel · 2024

Read Can Language Models Rival Mathematics Students? Evaluating Mathematical Reasoning through Textual Manipulation and Human Experiments by Andrii Nikolaiev, Yiannos Stathopoulos, Simone Teufel (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-11908.txt.

Introduction

Andrii Nikolaiev, Yiannos Stathopoulos, and Simone Teufel compared AI models with math students in a 2024 paper. The test ground was combinatorics, the branch of math about counting and arranging. The team built a set of 125 puzzle variants from twenty-five core problems.

Each problem came in five forms. Some had decoy details added, some had numbers changed, and some were worded in tangled ways. The math stayed the same, and the odd forms made it unlikely the models had memorized them. A GPT-4-based model beat the other models and, on number-changed variants, beat the human olympiad veterans. But rewording alone hurt the models a lot, while human scores barely moved. The authors say the gap between flexible human reasoning and brittle model reasoning is the real finding.

Catalog record

CREATORAndrii Nikolaiev, Yiannos Stathopoulos, Simone Teufel
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-11908

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Can Language Models Rival Mathematics Students? Evaluating Mathematical Reasoning through Textual Manipulation and Human Experiments Andrii Nikolaiev, Yiannos Stathopoulos, Simone Teufel Abstract In this paper we look at the ability of recent large language models (LLMs) at solving mathematical problems in combinatorics. We compare models LLaMA-2, LLaMA-3.1, GPT-4, and Mixtral against each other and against human pupils and undergraduates with prior experience in mathematical olympiads. To facilitate these comparisons we introduce the Combi-Puzzles dataset, which contains 125 problem variants based on 25 combinatorial reasoning problems. Each problem is presented in one of five distinct forms, created by systematically manipulating the problem statements through adversarial additions, numeric parameter changes, and linguistic obfuscation. Our variations preserve the mathematical core and are designed to measure the generalisability of LLM problem-solving abilities, while also increasing confidence that problems are submitted to LLMs in forms that have not been seen as training instances. We found that a model based on GPT-4 outperformed all other models in producing correct responses, and performed significantly better in the mathematical variation of the problems than humans. We also found that modifications to problem statements significantly impact the LLM’s performance, while human performance remains unaffected. 1 Introduction Mathematical problem-solving tasks are often presented as textual statements, requiring solutions that are logical, coherent, and culminate in a correct final answer. The task of solving mathematical word problems (MWP) has been a topic of interest in the fields of computational linguistics and artificial intelligence (AI) since the 1960s. Solving MWP as an area of research has seen a resurgence due partly to research questions afforded by new AI models, assessing the reasoning abilities of large language models (LLMs). Similar to human thinking patterns, efficient application of models usually requires accurately parsing text to identify relevant information, discard irrelevant details, and understand relationships between entities. Additionally, it requires creative reasoning to identify relevant arguments and steps, technical proficiency for performing routine calculations, and critical reasoning to examine and adjust the steps of an argument as mentioned by Bubeck et al. 2023 . Applying AI to solving advanced mathematical problems is now an open-challenge task for researchers, with challenges like AIMO Prize 1 1 1 https://aimoprize.com/ . being a notable example. Recent studies demonstrated that LLMs like GPT-4 exhibit human-like reasoning Wei et al. 2022 ; Achiam et al. 2024 and can tackle various natural language reasoning tasks. Kojima et al. 2023 found that GPT-4 performs well at reasoning tasks without further fine-tuning, but only experimented with problems requiring few reasoning steps to solve. Later, Collins et al. 2023 created a set of novel problems, presented them to GPT-4 and observed that the model has a tendency to over-rely on plausibly memorised examples or patterns. Therefore, whether the reasoning demonstrated by LLMs generalises is an open question that requires further investigation. This study examines the current abilities of established LLMs in mathematical reasoning using combinatorial problems with comparisons to the abilities of human participants. We present a fresh dataset and specialised methodology aimed at evaluating the generalisation of reasoning abilities in LLMs and investigate three questions: 1. Which LLM is the best at answering combinatorial reasoning problems? 2. Are humans better at reasoning on these problems than the best LLM? 3. Which problem variations affect LLMs and humans the most? 2 Background and Related work Most studies on language model evaluation in mathematical reasoning typically focus on mathematical word problems (MWPs). Datasets for this task are usually structured as mathematical statements and queries of mathematics-related concepts. Among the most widely utilised datasets are GSM8K Cobbe et al. 2021 , SVAMP Patel et al. 2021 with a focus on arithmetic reasoning, MMLU Hendrycks et al. 2021a – multiple choice question answering, and MATH Hendrycks et al. 2021b – word problem-solving. Some instances of combinatorial problems are included in the listed datasets. However, the emergence of LLMs has highlighted several issues: • Common datasets lack balanced difficulty levels, featuring either simple arithmetic or challenging olympiad-style problems. • Some datasets include test data from downstream tasks in the training data, causing pattern memorisation instead. • Few studies provide human evaluation of datasets, crucial for validating problem-solving abilities. To address these issues, several new datasets were introduced in recent years. Problems complexity. To enhance the difficulty level, the aforementioned datasets have been extended, and new datasets including challenging problems were introduced. Zheng et al. 2022 presented miniF2F, a dataset of formal Olympiad-level mathematics problem statements intended to provide a unified cross-system benchmark for neural theorem proving. Frieder et al. 2023 opted to create the GHOSTS dataset of graduate-level mathematics. Wang et al. 2024 introduced the MMLU-Pro dataset, including data from TheoremQA, featuring high-quality, human-annotated questions that necessitate the application of theorems for resolution; and SciBench, which includes advanced science questions derived from college exams, ensuring the inclusion of curriculum-aligned questions. In the PuzzleBench dataset, Mittal et al. 2024 shortlisted computationally challenging problems by manually scanning Wikipedia for diverse puzzles and NP-hard algorithmic problems. Additionally, the authors experimented with altering the configurations of certain problems to curate the complexity level of the tasks. Data contamination. Authors utilised GSM8k to replicate the questions in the dataset while withholding their corresponding parts and successfully identified contamination of the dataset within GPT-4 when guided instruction was used Golchin and Surdeanu 2024 . Zhang et al. 2024 highlighted contamination issues with existing datasets and mirrored the style and complexity of the GSM8k data to create their own dataset GSK1k using fresh problems contributed by human annotators. The models tested showed significantly worse results on the alternative problem set, which effectively highlights the issue of LLM’s memorisation of data. In the most recent research done Mirzadeh et al. 2024 , authors presented an updated GSM-symbolic dataset, an improved benchmark created from symbolic templates that allow for the generation of a diverse set of questions, including changing names, numbers or both, injection of irrelevant numerical information into the problem statements. The findings reveal that LLMs exhibit noticeable variance when responding to different instantiations of the same problem question. Human evaluation. Collins et al. 2023 conducted the first experiment with humans and LLMs in the domain of mathematics. The authors assessed the helpfulness of LLMs as mathematical assistants through direct interactions with undergraduate students and professors, and found notable instances of divergence between correctness and perceived helpfulness in LLM generations. In the work by Zheng et al. 2023 , authors found that LLMs show limitations in grading basic math problems which it is capable of solving. The test model, GPT-4, was able to solve the problem when asked separately, but it was misled by the provided answers, ultimately resulting in incorrect judgment. In the work by Bubeck et al. 2023 , the authors prompted the GPT-4 model with mathematical statements previously not seen online. From the experiments, the model demonstrates a high level of ability in choosing the right argument or path towards the solution but frequently fails to conduct correct reasoning due to arithmetic mistakes. Our goal is to evaluate the reasoning ability of established LLMs and draw comparisons with human subjects. We opted to create our dataset – the Combi-Puzzles dataset of combinatorial problems – with fresh problems addressing all issues above. First, we

Read the complete text