Assessment of Evolving Large Language Models in Upper Secondary Mathematics
1 st Mika Setälä, 2 nd Pieta Sikström*, 3 rd Ville Heilala, 4 th Tommi Kärkkäinen · 2025
Read Assessment of Evolving Large Language Models in Upper Secondary Mathematics by 1 st Mika Setälä, 2 nd Pieta Sikström*, 3 rd Ville Heilala, 4 th Tommi Kärkkäinen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-12347.txt.
Introduction
Finland's national matriculation exam is a stern test of school mathematics, and language models are now sitting it. Mika Setälä and co-authors assessed in 2025 how successive generations of models score on that exam.
Early models earned middling grades. Later models improved sharply, with some reaching near-perfect or perfect scores, matching top students and qualifying for university admission. The study tracks that rapid climb across model generations. The authors read the results as evidence that models can support learning and teaching, while noting that such proficiency also pressures how exams are designed.
Catalog record
| CREATOR | 1 st Mika Setälä, 2 nd Pieta Sikström*, 3 rd Ville Heilala, 4 th Tommi Kärkkäinen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-12347
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Assessment of Evolving Large Language Models in Upper Secondary Mathematics Thanks: Support from the Research Council of Finland under Grant number 353325 is gratefully acknowledged. 1 st Mika Setälä, 2 nd Pieta Sikström*, 3 rd Ville Heilala, 4 th Tommi Kärkkäinen Abstract Large language models (LLMs) have shown increasing promise in educational settings, yet their mathematical reasoning has been considered evolving. This study evaluates the mathematical capabilities of various LLMs using the Finnish matriculation examination, a high-stakes digital test for upper secondary education. Initial tests yielded moderate performance corresponding to mid-range grades, but later evaluations demonstrated substantial improvements as the language models evolved. Remarkably, some models achieved near-perfect or perfect scores, matching top student performance and qualifying for university admission. Our findings highlight the rapid advances in the mathematical proficiency of LLMs and illustrate their potential as underlying tools to support learning and teaching in a variety of ways. I Introduction Conversational learning technologies have demonstrated significant potential in educational contexts across a wide range of disciplines [ 1 , 2 , 3 ] . Since OpenAI launched ChatGPT in late 2022, the education landscape has undergone disruption, where Large Language Model (LLM) based learning tools have been increasingly embedded in education. LLMs can enhance personalized learning [ 4 ] and provide individual scaffolding for students while enhancing engagement through interactive learning experiences [ 5 ] . The findings from a literature review [ 6 ] suggested that, in general, ChatGPT can enhance personalized learning, motivation, and engagement. However, also critical views have been expressed on whether the summarization capabilities of generative AI are used to outsource learning to a machine instead of scaffolding student’s own learning [ 7 ] . The potential benefits of LLMs are also present in mathematics, where several studies have concluded the benefits of using conversational AI [ 8 , 9 , 10 , 11 , 12 ] . For example, a conversational agent can both answer mathematical questions and also serve as a mathematical search engine and knowledge base [ 8 ] . With the growing recognition of the benefits and increasing prevalence of LLMs, a deeper understanding of their role in education is essential [ 13 ] . In particular, given their somewhat ambiguous capabilities in mathematical reasoning [ 14 ] , it is crucial to explore their impact and effectiveness in mathematics learning. Digital performance in Finland is distinguished 1 1 1 Digital Economy and Society Index 2022 - Country Reporting , and nearly 80% of the learning materials in upper secondary education were digital in 2023 [ 15 ] . The national matriculation examination 2 2 2 The Finnish Matriculation Examination taken at the end of the Finnish upper secondary school (ISCED Level 3) is a fully digitalized, biannual, high-stakes test designed to assess how well students have met curriculum requirements [ 16 ] . Since rigorous benchmarking of LLMs relies on standardized evaluation criteria to ensure fair and objective comparison between models, the aim of this study was to contribute to the understanding of LLMs’ mathematical reasoning capabilities by using the Finnish matriculation examination in advanced mathematics as test data. To explore the mathematical performance of LLMs, we pose the following research question: How do different LLM models perform in Finnish matriculation examination? II Capabilities of LLMs in mathematics TABLE I: Versions and initial release dates of OpenAI a , Google b , and DeepSeek c models LLM Model Initial Release Date Initial Knowledge Cutoff Description ChatGPT (GPT-3.5) a 2022/11/30 2021/09 The initial OpenAI ChatGPT release, a sibling model to InstructGPT GPT-4 a 2023/03/14 2021/09 A large multimodal model introduced with the ChatGPT Plus subscription Bard b 2023/03/21 Google’s chatbot initially based on the LaMDA language model. DeepSeek Coder c 2023/11/01 Initial release focusing on coding capabilities GPT-4 Turbo a 2023/11/06 2023/04 A model with a 128k context window DeepSeek LLM c 2023/11/29 Expanded language model with improved performance Gemini 1 b 2023/12/06 Bard rebranded to Gemini and updated to the Gemini LLM DeepSeek-MoE c 2024/01/09 Mixture-of-Experts architecture for more efficient modeling Gemini 1.5 b 2024/02/15 A new architecture and expanded context handling DeepSeek V2 c 2024/05/06 Introduced new features and enhancements GPT-4o a 2024/05/13 2023/10 Multimodal model capable of processing text, images, and audio o1-preview a 2024/09/12 2023/10 Chain-of-thought reasoning model o1-mini a 2024/09/12 2023/10 Tunable reasoning effort for STEM fields, esp. in math and coding o1 a 2024/12/05 2023/10 A state-of-the-art reasoning model for problem-solving in various domains Gemini 2.0 Flash Exp b 2024/12/11 Enhancements in speed and performance DeepSeek V3 c 2024/12/25 Performance improvements with multimodal support DeepSeek R1 c 2025/01/20 Logical inference and reasoning for mathematical problem-solving Gemini 2.0 Flash b 2025/01/30 Adjusted reasoning with large context window o3-mini a 2025/01/31 2023/10 A reasoning model particularly for STEM subjects in science, math, and coding The rapid adoption of LLMs has fueled competition among providers, leading to an unprecedented acceleration in development. In 2023 alone, 21 major LLMs were released [ 17 ] , and the speed has been increasing ever since. Table I presents an overview of the development trajectory and different versions of some popular LLMs. Experiences and experiments on the use of LLMs in mathematics education are conditioned on the mathematical capabilities of these systems. For example, ChatGPT’s mathematical competence is argued to be sufficient to teach rational numbers according to middle school curricula [ 18 ] . However, it lacked pedagogical content knowledge, particularly in explaining the representation of rational numbers on a number line and their conversion to decimal form. Some argued that the use of ChatGPT in mathematical reasoning requires expertise in calculus concepts [ 10 ] . In addition, in engineering mathematics, certain challenges necessitated the adaptation of teaching strategies and methodologies [ 19 ] . [ 8 ] reported that the exam-solving capabilities of ChatGPT/GPT-4 were well below the level of a graduate student. In addition, according to tests by [ 20 ] in January 2023, ChatGPT was not able to produce high-quality proofs or calculations consistently, even if the quality of the answers was sometimes surprisingly good. These observations were reflected by [ 11 ] , who concluded that mathematical reasoning and coding abilities should be enhanced by training with more mathematical texts and code data. In October 2024, Apple researchers [ 21 ] presented findings that LLMs fail to perform genuine logical reasoning in mathematical tasks. Their study revealed that even minor modifications in the formulation of a problem could significantly degrade model performance. Previous assessments of AI mathematical proficiency have predominantly relied on three key datasets (Table II ). MATH Dataset [ 22 ] is a collection of 12,500 problems sourced from high school and undergraduate mathematics competitions. The subset MATH-500 comprises the 500 most challenging problems, widely used to evaluate LLM performance in mathematical reasoning. American Invitational Mathematics Examination (AIME) [ 23 ] is a 15-question, three-hour test that has been administered to students scoring in the top 5% of the AMC 12 competition. AIME serves as an intermediate stage in the selection process for the USAMO and the USAJMO, which are key qualifiers for the International Mathematical Olympiad. The difficulty of AIME problems increases progressively, and each answer is a three-digit integer. The problems