Is GPT-OSS Good? A Comprehensive Evaluation of OpenAI’s Latest Open Source Models

Ziqian Bi 1,2* , Keyu Chen 1,3* , Chiung-Yi Tseng 1,4* , Danyang Zhang 1,5* , Tianyang Wang 1 , Hongying Luo 1 , Lu Chen 1 , Junming Huang 1 , Jibin Guan 6 , Junfeng Hao 6 , Junhao Song 7† · 2025

Read Is GPT-OSS Good? A Comprehensive Evaluation of OpenAI’s Latest Open Source Models by Ziqian Bi 1,2* , Keyu Chen 1,3* , Chiung-Yi Tseng 1,4* , Danyang Zhang 1,5* , Tianyang Wang 1 , Hongying Luo 1 , Lu Chen 1 , Junming Huang 1 , Jibin Guan 6 , Junfeng Hao 6 , Junhao Song 7† (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-12461.txt.

Introduction

Ziqian Bi and co-authors present a 2025 evaluation of OpenAI's open-weight language models. The two models, released in August 2025, are the firm's first open releases since 2019. They hold 120 and 20 billion parameters in sparse designs. The team pitted both against six rival open models across ten benchmarks, from knowledge and math to code and chat.

Oddly, the smaller model beat its big sibling on several tests, code and knowledge among them. It also used far less memory and energy per reply. Both models landed mid-pack overall. Code was a strength, while multilingual tasks lagged. The authors warn that sparse scaling may not pay off in proportion. Test scripts are posted on the project page.

Catalog record

CREATORZiqian Bi 1,2* , Keyu Chen 1,3* , Chiung-Yi Tseng 1,4* , Danyang Zhang 1,5* , Tianyang Wang 1 , Hongying Luo 1 , Lu Chen 1 , Junming Huang 1 , Jibin Guan 6 , Junfeng Hao 6 , Junhao Song 7†
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-12461

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Is GPT-OSS Good? A Comprehensive Evaluation of OpenAI’s Latest Open Source Models Thanks: ∗ * : Equal contribution. † \dagger : Corresponding author: Junhao Song ([email protected]) Ziqian Bi 1,2* , Keyu Chen 1,3* , Chiung-Yi Tseng 1,4* , Danyang Zhang 1,5* , Tianyang Wang 1 , Hongying Luo 1 , Lu Chen 1 , Junming Huang 1 , Jibin Guan 6 , Junfeng Hao 6 , Junhao Song 7† Abstract In August 2025, OpenAI released GPT-OSS models, its first open weight large language models since GPT-2 in 2019, comprising two mixture of experts architectures with 120B and 20B parameters. We evaluated both variants against six contemporary open source large language models ranging from 14.7B to 235B parameters, representing both dense and sparse designs, across ten benchmarks covering general knowledge, mathematical reasoning, code generation, multilingual understanding, and conversational ability. All models were tested in unquantised form under standardised inference settings, with statistical validation using McNemar’s test and effect size analysis. Results show that gpt-oss-20B consistently outperforms gpt-oss-120B on several benchmarks, such as HumanEval and MMLU, despite requiring substantially less memory and energy per response. Both models demonstrate mid-tier overall performance within the current open source landscape, with relative strength in code generation and notable weaknesses in multilingual tasks. These findings provide empirical evidence that scaling in sparse architectures may not yield proportional performance gains, underscoring the need for further investigation into optimisation strategies and informing more efficient model selection for future open source deployments. I Introduction The past five years have seen a rapid expansion of open source large language models (LLMs), with dense architectures such as Llama and Gemma joined by increasingly capable mixture of experts (MoE) designs. MoE architectures activate only a fraction of parameters per token [ 1 , 2 ] , offering a theoretical path to improved compute efficiency and scalability [ 3 , 4 , 5 ] . In August 2025, OpenAI released GPT OSS models, its first open weight models since GPT-2 in 2019 [ 6 ] , introducing MoE variants with 20B and 120B total parameters into a landscape already shaped by competitive offerings from Meta [ 7 ] , Google [ 8 ] , DeepSeek [ 9 ] , Alibaba [ 10 , 11 ] and Microsoft [ 12 ] . Fig. 1: Multi-dimensional performance comparison across eight evaluated models. GPT-OSS models (highlighted) show middle-tier performance with particular strength in code generation but weakness in multilingual tasks. While scaling laws have been widely used to guide LLM development, empirical studies on their validity for sparse architectures remain limited. Existing evaluations have often focused on single domains or isolated benchmarks [ 13 , 14 ] , leaving open the question of whether larger MoE variants consistently outperform smaller ones across diverse capabilities. This gap matters both for understanding the training dynamics of sparsely activated networks and for making informed deployment choices under resource constraints. To address this gap, we position GPT-OSS alongside six contemporary open source LLMs spanning 14.7B to 235B parameters, covering both dense and sparse architectures. We design a controlled evaluation across ten established benchmarks grouped into five capability domains: general understanding, mathematical reasoning, code generation, multilingual comprehension, and conversational ability. This framework enables a direct comparison of performance and efficiency characteristics, isolating the relationship between parameter scale, architecture type and task outcomes. Our analysis serves two purposes. From a research perspective, it tests whether observed performance trends align with established scaling expectations in the MoE setting. From a practical perspective, it examines the cost performance profile of GPT-OSS relative to strong open source baselines, incorporating memory, latency, and energy measurements that are often omitted from academic reports. The results, presented in the following sections, provide evidence relevant to both theoretical modelling of scaling behaviour and the design of efficient deployment strategies in the evolving open source community. II Related Work II-A Large Language Models Evaluation The evaluation of large language models has evolved significantly since the introduction of transformer architectures. Early work by Radford et al. [ 6 ] laid the foundation for systematic LLM evaluation through the development of benchmarks like WebText and subsequent iterations. The field has since developed comprehensive evaluation frameworks addressing multiple dimensions of model capabilities. Hendrycks et al. [ 15 ] introduced MMLU (Massive Multitask Language Understanding) as a comprehensive benchmark spanning 57 subjects, providing broad coverage of human knowledge domains. This benchmark has become a standard metric for assessing general understanding capabilities, though recent work highlights potential limitations including dataset contamination and the need for more challenging evaluation scenarios. The evaluation of mathematical reasoning capabilities has been extensively studied through benchmarks like GSM8K, introduced by Cobbe et al. [ 16 ] , which tests multi-step problem-solving abilities. Subsequent work by Lewkowycz et al. [ 17 ] with Minerva demonstrated that specialised training on mathematical content can significantly improve performance, raising questions about the generalisation of reasoning capabilities across domains. II-B Code Generation Assessment The HumanEval benchmark, developed by Chen et al. [ 18 ] , established a rigorous framework for evaluating code generation capabilities through functional correctness testing. This work has been extended by Austin et al. [ 19 ] with MBPP (Mostly Basic Python Problems), and by Li et al. [ 20 ] with additional multilingual code generation benchmarks. Recent studies by Nijkamp et al. [ 21 ] demonstrate that code-specific pretraining significantly impacts performance, suggesting that general purpose models may face inherent limitations in programming tasks. The emergence of specialised coding models like Codex [ 18 ] and CodeGen [ 21 ] has prompted renewed interest in understanding the relationship between model architecture and code generation capabilities. Fried et al. [ 22 ] showed that instruction-tuned models can achieve competitive performance with significantly fewer parameters, challenging assumptions about scale requirements for code generation tasks. II-C Multilingual Model Evaluation Cross-lingual evaluation has gained prominence with the increasing deployment of LLMs in global contexts. The C-Eval benchmark, introduced by Huang et al. [ 23 ] , provides comprehensive coverage of Chinese language understanding across multiple disciplines. Similar efforts include MMLU variants for other languages and the XTREME benchmark suite for cross-lingual transfer evaluation. Research by Winata et al. [ 24 ] demonstrates that multilingual capabilities often come at the cost of monolingual performance, a phenomenon termed the “curse of multilinguality.” This trade-off is particularly relevant for models like GPT-OSS that aim for broad language coverage without language-specific optimisation. II-D Mixture of Experts Architectures The mixture of experts paradigm has experienced renewed interest with models like Switch Transformer by Fedus et al. [ 2 ] and GLaM by Du et al. [ 25 ] , demonstrating that sparse models can achieve competitive performance with reduced computational requirements. The routing mechanisms and expert specialisation patterns in these models have been extensively studied by Lewis et al. [ 26 ] , revealing complex dynamics between model capacity and task performance. Recent work, such as the branch train merge approach from Artetxe et al. [ 27 ] and the Expert Choice routing from Zhou et al. [ 28 ] , has advanced our understanding of optimal MoE configurations. These studies provide context for understanding the performance characteristics of GPT-OSS’s MoE implementation, particularly the unexpected inverse scaling observed between the

Read the complete text