Cross-Platform Evaluation of Reasoning Capabilities in Foundation Models
J. de Curtò, I. de Zarzà, Pablo García, Jordi Cabot · 2025
Read Cross-Platform Evaluation of Reasoning Capabilities in Foundation Models by J. de Curtò, I. de Zarzà, Pablo García, Jordi Cabot (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-26732.txt.
Introduction
J de Curtò, I de Zarzà, Pablo García, and Jordi Cabot compare the reasoning of foundation models across three kinds of computing platform. The same model may behave differently on a supercomputer, a cloud service, or a university cluster, which would cloud any benchmark. Their test bed spans the MareNostrum 5 supercomputer, Nebius AI Studio, and a university node with eight H200 graphics cards.
The four authors run fifteen models on seventy-nine problems across eight academic domains, in three phases. Early phases set a baseline on nineteen problems and check that results reproduce across platforms. The full run then probes generalisation at scale. Their findings push against simple scaling beliefs: training data quality mattered more than model size. They close with practical guidelines for picking models in education, production, and research settings.
Catalog record
| CREATOR | J. de Curtò, I. de Zarzà, Pablo García, Jordi Cabot |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-26732
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Cross-Platform Evaluation of Reasoning Capabilities in Foundation Models J. de Curtò, I. de Zarzà, Pablo García, Jordi Cabot Abstract This paper presents a comprehensive cross-platform evaluation of reasoning capabilities in contemporary foundation models, establishing an infrastructure-agnostic benchmark across three computational paradigms: HPC supercomputing (MareNostrum 5), cloud platforms (Nebius AI Studio), and university clusters (a node with eight H200 GPUs). We evaluate 15 foundation models across 79 problems spanning eight academic domains (Physics, Mathematics, Chemistry, Economics, Biology, Statistics, Calculus, and Optimization) through three experimental phases: (1) Baseline establishment: Six models (Mixtral-8x7B, Phi-3, LLaMA 3.1-8B, Gemma-2-9b, Mistral-7B, OLMo-7B) evaluated on 19 problems using MareNostrum 5, establishing methodology and reference performance; (2) Infrastructure validation: The 19-problem benchmark repeated on university cluster (seven models including Falcon-Mamba state-space architecture) and Nebius AI Studio (nine state-of-the-art models: Hermes-4 70B/405B, LLaMA 3.1-405B/3.3-70B, Qwen3 30B/235B, DeepSeek-R1, GPT-OSS 20B/120B) to confirm infrastructure-agnostic reproducibility; (3) Extended evaluation: Full 79-problem assessment on both university cluster and Nebius platforms, probing generalization at scale across architectural diversity. Results challenge prevailing scaling assumptions through a parameter efficiency paradox: Hermes-4-70B (70B parameters) achieves the highest score among extended models (0.598), outperforming both its 405B counterpart (0.573) and Meta’s LLaMA 3.1-405B (0.560). Domain-specific analysis reveals LLaMA 3.1-405B achieves record Calculus performance (0.717), while DeepSeek-R1 sets unprecedented standards for reasoning transparency (0.716 step-accuracy). Qwen3 models demonstrate exceptional consistency (0.013 score variance, 3 × \times better than alternatives). We identify a fundamental transparency-correctness trade-off: DeepSeek-R1’s high step-accuracy (0.716) correlates weakly with final answers (r=0.249), whereas Qwen3 exhibits near-zero correlation (r=0.095), suggesting “shortcut learning” that bypasses explicit reasoning chains. Longitudinal comparison (2024 vs 2025) reveals domain-specific evolution: Calculus improved dramatically from mid-tier to top-ranked domain (+24.7%), while Optimization remains universally challenging across model generations (+4.7% improvement only). Cross-model disagreement analysis identifies Physics kinematics as the most controversial domain (std dev 0.335). Infrastructure validation across three platforms confirms reasoning quality is model-intrinsic: performance remains consistent across HPC (MareNostrum 5), cloud (Nebius), and university cluster environments (¡3% variance: LLaMA-3.1-8B − 2.9 % -2.9\% , Phi-3-mini − 1.1 % -1.1\% ), democratizing rigorous evaluation beyond supercomputing institutions. University cluster validation reveals competitive performance of non-transformer architectures (Falcon-Mamba state-space model achieves 0.590, matching transformer baseline LLaMA-3.1-8B at 0.576) and demonstrates that dense smaller models (Phi-4-mini, 14B) outperform larger sparse MoE architectures (Phi-3.5-MoE, 42B: 0.674 vs 0.569). These findings challenge conventional scaling assumptions, establish training data quality as more critical than model size, and provide actionable guidelines for model selection across educational, production, and research contexts. The tri-infrastructure methodology and 79-problem benchmark enable longitudinal tracking of reasoning capabilities as foundation models evolve. Cross-Platform Evaluation of Reasoning Capabilities in Foundation Models [type=author, orcid=0000-0002-8334-4719] [type=author, orcid=0000-0002-5844-7871] [type=author,] [type=author, orcid=0000-0003-2418-2489] keywords Foundation Models ,Large Language Models ,Reasoning Evaluation ,Artificial Intelligence ,Supercomputing ,Cloud AI Infrastructure † † credit: Conceptualization, Data Curation, Investigation, Methodology, Software, Supervision, Validation, Visualization, Writing - Original Draft, Writing - Review & Editing † † credit: Conceptualization, Data Curation, Formal Analysis, Investigation, Methodology, Software, Validation, Visualization, Writing - Review & Editing † † credit: Validation, Software † † credit: Conceptualization, Supervision, Validation, Writing - Review & Editing † † corresponding: Corresponding author. 1 Introduction Large language models (LLMs) have rapidly transformed artificial intelligence by extending natural language understanding and reasoning into complex analytical tasks. While prior work has assessed general comprehension and task performance, the cross-domain reasoning consistency of these systems remains poorly understood. We evaluate 15 foundation models across 79 problems spanning eight academic domains (Physics, Mathematics, Chemistry, Economics, Biology, Statistics, Calculus, and Optimization) through three complementary experimental phases: (1) Baseline establishment: Six models (Mixtral-8x7B, Phi-3, LLaMA 3.1-8B, Gemma-2-9b, Mistral-7B, OLMo-7B) evaluated on 19 problems using MareNostrum 5, establishing methodology and reference performance; (2) Infrastructure validation: Seven models including non-transformer architectures (Falcon-Mamba state-space model) on university cluster and nine state-of-the-art models—Hermes-4 (70B and 405B), LLaMA 3.1-405B and 3.3-70B, Qwen3 (30B and 235B), DeepSeek-R1, and GPT-OSS (20B and 120B)—on Nebius cloud platform, both using the 19-problem set to confirm infrastructure-agnostic reproducibility; (3) Extended evaluation: Full 79-problem benchmark on university infrastructure and Nebius platform, probing generalization at scale. This tri-infrastructure design allows us to validate results under distinct computational environments—supercomputing (MareNostrum 5), cloud platforms (Nebius AI Studio), and university clusters—thereby maximizing external validity and reproducibility while democratizing access to rigorous reasoning evaluation beyond specialized HPC facilities. The remainder of this article is organized as follows. Section 2 reviews the evolution of foundation models, reasoning evaluation benchmarks, and cross-domain evaluation methodologies, contextualizing our work within the broader AI research landscape. Section 3 presents the initial baseline evaluation conducted on MareNostrum 5 supercomputer, establishing the evaluation methodology and reference performance metrics for six foundation models across 19 problems. Section 4 describes the expansion from the original 19-problem baseline to the comprehensive 79-problem benchmark, detailing domain coverage, difficulty stratification, and problem selection criteria. Section 5 formalizes our evaluation framework, including semantic similarity metrics, step-wise accuracy measurement, and consistency quantification across computational platforms. Section 6 presents an infrastructure-agnostic validation study on the 19-problem benchmark, evaluating seven models (including legacy architectures and state-space models) across university cluster and comparing results with the MareNostrum 5 baseline to establish reproducibility with ¡3% variance across computational platforms, as well as using nine additional state-of-the-art models on Nebius AI Studio infrastructure, including large-scale variants (Hermes-4-405B, LLaMA 3.1-405B, Qwen3-235B). Section 7 extends the evaluation to 79 problems enabling comprehensive analysis of domain-specific performance, the parameter efficiency paradox, and the transparency-correctness trade-off at scale. Section 8 synthesizes findings across all three infrastructures (MareNostrum 5, Nebius AI Studio, and university cluster), analyzing longitudinal evolution patterns, cross-model disagreement, and architectural diversity across 15 foundation models. Finally, Section 9 consolidates key findings, discusses practical implications for model selection across educational, production, and research contexts, and outlines future research directions for advancing reasoning-capable AI systems. 2 Background and Related Work LLMs have evolved through successive generations of transformer-based architectures ( Vaswani et al. 2017 ) , including BERT ( Devlin et al. 2018 ) , GPT-3 ( Brown et al. 2020 ) , and PaLM ( Chowdhery et al. 2022 ) . More recent work has emphasized efficiency (e.g., Phi-3 ( Abdin et al. 2024 ) , Gemma ( Team et al. 2024 ) ) and mixture-of-experts architectures (e.g., Mixtral ( Mistral AI 2023 ) ). Reasoning-oriented evaluations—ARC ( Clark et al. 2018 ) , MMLU ( Hendrycks et al. 2021 ) , and chain-of-thought prompting ( Wei et al. 2022b ) —have demonstrated that explicit reasoning steps improve performance but not necessarily correctness or consistency. Our framework extends these evaluations by systematically measuring semantic similarity and step-wise correctness across domains and infrastructures. The trajectory from early transformer models to contemporary foundation models represents one of the most rapid capability escalations in AI history. ( Kaplan et al. 2020 ) established power-law relationships between model scale, data size, and performance, predicting that larger models trained on more data would consistently improve. This motivated the development of massive models like GPT-3 ( Brown et al. 2020 ) (175B parameters) and PaLM ( Chowdhery et al. 2022 ) (540B parameters). However, ( Wei et al. 2022a ) documented emergent capabilities —abilities that appear suddenly at certain scale thresholds rather than gradually improving. Reasoning abilities, particularly multi-step logical deduction, were identified as one such