U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in Large Language Models
Konstantin Chernyshev, Vitaliy Polshkov, Ekaterina Artemova, Sergei Tilga, Alex Myasnikov, Vlad Stepanov, Alexei Miasnikov · 2024
Read U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in Large Language Models by Konstantin Chernyshev, Vitaliy Polshkov, Ekaterina Artemova, Sergei Tilga, Alex Myasnikov, Vlad Stepanov, Alexei Miasnikov (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03205.txt.
Introduction
Chernyshev and co-authors release in 2024 a benchmark of university level math problems for testing large language models. Existing tests are small, stick to school level problems, or lack topic range, and problems with figures are barely covered. Their set holds 1,100 unpublished open ended problems drawn from teaching materials, balanced across six core subjects, with a fifth of them using figures.
Because answers are open ended, a language model judges whether solutions are right. A companion data set tests how well models can do that judging. Top models scored high on text problems but far lower on visual ones, and judging itself proved hard even for the best models. The results expose clear limits in visual math reasoning.
Catalog record
| CREATOR | Konstantin Chernyshev, Vitaliy Polshkov, Ekaterina Artemova, Sergei Tilga, Alex Myasnikov, Vlad Stepanov, Alexei Miasnikov |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-03205
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in Large Language Models Konstantin Chernyshev, Vitaliy Polshkov, Ekaterina Artemova, Sergei Tilga, Alex Myasnikov, Vlad Stepanov, Alexei Miasnikov Abstract Current evaluations of mathematical skills in Large Language Models are constrained by benchmarks lacking scope, particularly for multi-modal problems — frequently relying on school-level ( Cobbe et al., 2021 ; Lu et al., 2023 ; Zhang et al., 2024 ) , niche Olympiad-style ( Fang et al., 2024 ; Mao et al., 2024 ) , simple quiz format ( Yue et al., 2023 ; Qiao et al., 2024 ) or relatively small ( Lewkowycz et al., 2022 ) datasets. To address this, we introduce U-MATH , a novel benchmark comprising 1,100 unpublished open-ended university-level problems sourced from current US curricula, with 20% incorporating visual elements. Given the free-form nature of U-MATH problems, we employ LLM judges for solution evaluation and release 𝝁 \boldsymbol{\mu} -MATH , a meta-evaluation benchmark composed of 1,084 U-MATH-derived tasks enabling precise assessment of these judges. Benchmarking leading LLMs reveals marked limitations in multi-modal reasoning, with maximum accuracy reaching 93.1% on textual tasks but only 58.5% on visual ones. Furthermore, solution judgment proves challenging, requiring the most advanced models to achieve meaningfully high performance, even still peaking at an imperfect F1-score of 90.1%. We open-source U-MATH, 𝝁 \boldsymbol{\mu} -MATH, and all our evaluation code. 1 1 1 https://github.com/toloka/u-math Example: Differential Calculus. U-MATH Problem: The function s(t)=2\cdot t^{3}-3\cdot t^{2}-12\cdot t+8 represents the position of a particle traveling along a horizontal line. 1. Find the velocity and acceleration functions. 2. Determine the time intervals when the object is slowing down or speeding up. Reference Solution (shortened): The velocity is v(t)=s^{\prime}(t)=\boxed{6\cdot t^{2}-6\cdot t-12} , zeros of the v(t) are t=-1,2 . The acceleration is a(t)=v^{\prime}(t)=\boxed{12\cdot t-6} , zero of the a(t) is t=\frac{1}{2} . It speeds up when v(t) and a(t) have the same sign, and slows down when opposite. \begin{array}[]{cccc}\hline\cr\text{Interval}&v(t)&a(t)&\text{Behavior}\\ \hline\cr(-\infty,-1)&>0&<0&\text{Slowing down}\\ (-1,\frac{1}{2})&<0&<0&\text{Speeding up}\\ (\frac{1}{2},2)&<0&>0&\text{Slowing down}\\ (2,\infty)&>0&>0&\text{Speeding up}\\ \hline\cr\end{array} Accounting for non-negative time, speed up on \boxed{(0,\nicefrac{{1}}{{2}})\text{ and }(2,\infty)} , slow down on \boxed{(\nicefrac{{1}}{{2}},2)} . Figure 1: A U-MATH sample. A common students’ error reported by the author is overlooking time non-negativity. 1 Introduction Assessing the mathematical proficiency of Large Language Models (LLMs) is crucial for evaluating their fundamental reasoning capabilities ( Ahn et al., 2024 ) . The most widely used benchmarks, GSM8K ( Cobbe et al., 2021 ) and MATH ( Hendrycks et al., 2021 ) , primarily cover school-level problems, overlooking advanced topics and facing rapid saturation ( Achiam et al., 2023 ) . Although some MATH problems and other recent works introduce harder concepts, they are limited in size and scope, relying on competition-style problems and neglecting the practical middle-ground of university-level coursework. There is also growing demand for visual reasoning assessment in multi-modal LLMs ( Ahn et al., 2024 ) . Datasets such as the recent MATH-V ( Wang et al., 2024a ) provide numerous visual problems but face similar topic limitations or rely on the multiple-choice format, making the tasks significantly easier ( Li et al., 2024b ; Pezeshkpour and Hruschka, 2023 ) . In turn, reliably evaluating complex free-form responses is challenging ( Hendrycks et al., 2021 ) , which results in LLM judges becoming the de facto standard despite known biases and inconsistencies ( Zheng et al., 2023 ) . These biases are often overlooked and unquantified, preventing potential correction. Quantifying auto-evaluation errors requires datasets designed specifically to assess the evaluators themselves, also called meta-evaluations. While mathematical meta-evaluation datasets do exist, they are mostly based on GSM8K and MATH, inheriting their scope limitations. To address these gaps, we introduce the U-MATH ( U niversity Math ) and \boldsymbol{\mu} -MATH ( M eta U - MATH ) benchmarks. Our main contributions are: 1. U-MATH (Section 3 ): We open-source 1,100 university-level problems, balanced across six core university subjects. The problems are collected from actual coursework and supplied with correct answers, with approximately 20% incorporating visual elements. 2. \boldsymbol{\mu} -MATH (Section 3.3 ): We introduce a set of 1084 meta-evaluation tasks designed to assess the quality of LLM judges by selecting approximately 25% of the U-MATH problems, supplying each with four solutions produced by four different top-performing language models, and providing ground truth labels on generated solutions’ correctness. 3. Comparative analysis (Section 4 ): We compare various open-source and proprietary LLMs on U-MATH and \boldsymbol{\mu} -MATH, revealing significant deficiencies in solving university-level multi-modal problems. We also find proprietary models to outperform open-source ones on these tasks, while near-parity is observed with the text modality. Judgment also proves challenging for LLMs, with only the best-performing and most recent models attaining adequately high scores. In addition, we demonstrate that most current systems exhibit biased and unstable judgment performance. Finally, we establish that judgment as a skill is distinct from problem-solving and identify characteristic behavioral tendencies in LLM judges. We release the U-MATH and \boldsymbol{\mu} -MATH benchmarks under a permissive license to facilitate further research and ensure reproducibility. 2 Background Evaluating mathematical capabilities of LLMs is an essential direction of AI research ( Ahn et al., 2024 ) . Apart from mathematical proficiency being important in and of itself, studies show that fine-tuning with math and code-related data enhances models’ fundamental ‘cognitive skills’ ( Prakash et al., 2024 ) and reasoning capabilities ( Chen et al., 2024 ) , further necessitating the creation of mathematical evaluation datasets. Despite significant progress, many existing datasets are limited in scope, complexity of the problems, or size, as evidenced by the summary in Table 1 . Dataset Levels %Uni. Level #Test %Visual %Free-form #Free-form Text-only #Free-form Visual Uni. Level Test Uni. Level Test MMLU {}_{\textit{Math}} ( Hendrycks et al., 2020 ) E H C 0 1.3k 0 0 0 0 GSM8k ( Cobbe et al., 2021 ) E 0 1k 0 0 0 0 MATH ( Hendrycks et al., 2021 ) H O 0 5k 0 100 0 0 MiniF2F ( Zheng et al., 2021 ) E H O 0 244 0 100 0 0 OCWCourses ( Lewkowycz et al., 2022 ) U 100 272 0 100 272 0 ProofNet ( Azerbayev et al., 2023 ) C U \approx 50 371 0 100 \approx 180 0 CHAMP ( Mao et al., 2024 ) H O 0 270 0 100 0 0 MathOdyssey ( Fang et al., 2024 ) H U O \approx 25 387 0 100 \approx 50 0 MMMU {}_{\textit{Math}} ( Yue et al., 2023 ) C 0 505 100 0 0 0 MathVista ( Lu et al., 2023 ) E H C 0 5k 100 46 0 0 MATH-V ( Wang et al., 2024a ) E H O 0 3k 100 50 0 0 We-Math ( Qiao et al., 2024 ) E H U \approx 20 1.7k 100 0 0 0 MathVerse ( Zhang et al., 2024 ) H 0 4.7k 83.3 45 0 0 U-MATH (this work) U 100 1.1k 20 100 900 200 Table 1: Existing auto-evaluated math benchmarks along with their sizes, visual sample percentages, and open-ended problem percentages. Level markers: E Elementary to Middle School, H High School, C College, U University, O Olympiads. Textual Mathematical Benchmarks. Datasets like MathQA ( Amini et al., 2019 ) and