COMPASS: A Multi-Dimensional Benchmark for Evaluating Code Generation in Large Language Models
James Meaden · 2025
Read COMPASS: A Multi-Dimensional Benchmark for Evaluating Code Generation in Large Language Models by James Meaden (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-13757.txt.
Introduction
Code tests for AI models usually check only whether the program works, ignoring whether it is fast or clean. James Meaden and co-authors build a wider test in a 2025 paper. Their benchmark, COMPASS, holds fifty contest problems from real coding competitions. Nearly four hundred thousand human entries set the baseline. Three top reasoning models were graded on correctness, speed, and quality at once.
High correctness did not guarantee fast or tidy code, the results show. Standard industry tools judged the quality side. Slow code that passes tests is marked down here, unlike older benchmarks. The authors pitch COMPASS as a guide toward production-ready code models.
Catalog record
| CREATOR | James Meaden |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-13757
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
COMPASS: A Multi-Dimensional Benchmark for Evaluating Code Generation in Large Language Models Thanks: All authors are affiliated with Codility. Corresponding author: James Meaden; [email protected] James Meaden Abstract Current code generation benchmarks focus primarily on functional correctness while overlooking two critical aspects of real-world programming: algorithmic efficiency and code quality. We introduce COMPASS (COdility’s Multi-dimensional Programming ASSessment), a comprehensive evaluation framework that assesses code generation across three dimensions: correctness, efficiency, and quality. COMPASS consists of 50 competitive programming problems from real Codility competitions, providing authentic human baselines from 393,150 submissions. Unlike existing benchmarks that treat algorithmically inefficient solutions identically to optimal ones provided they pass test cases, COMPASS systematically evaluates runtime efficiency and code quality using industry-standard analysis tools. Our evaluation of three leading reasoning-enhanced models, Anthropic Claude Opus 4, Google Gemini 2.5 Pro, and OpenAI O4-Mini-High, reveals that models achieving high correctness scores do not necessarily produce efficient algorithms or maintainable code. These findings highlight the importance of evaluating more than just correctness to truly understand the real-world capabilities of code generation models. COMPASS serves as a guiding framework, charting a path for future research toward AI systems that are robust, reliable, and ready for production use. I Introduction Large language models (LLMs) have achieved remarkable proficiency in code generation, yet current evaluation benchmarks predominantly assess only functional correctness through test case execution [ 1 , 2 , 3 ] . This narrow focus fails to capture essential aspects of real-world software development where code must not only work correctly but also perform efficiently and maintain high quality standards. Existing benchmarks such as HumanEval [ 1 ] , MBPP [ 2 ] , and HackerRank-ASTRA [ 4 ] evaluate models based on functional correctness, measuring whether generated code produces expected outputs for given test cases. While these benchmarks have been valuable for measuring code generation capabilities, they overlook critical aspects of professional programming: the ability to write code that is maintainable and runs efficiently. In real-world software development, a solution that passes functional tests but runs in exponential time or violates coding best practices would be considered inadequate, yet current benchmarks would score it as perfect. This evaluation gap is problematic given the increasing deployment of LLMs in production environments where code efficiency and quality directly impact user experience, system security and scalability, and long-term maintainability [ 5 , 6 ] . Recent reasoning-enhanced models demonstrate sophisticated problem-solving capabilities, yet the field lacks a comprehensive understanding of whether this enhanced reasoning translates to better algorithmic efficiency and code quality. To address these limitations, we introduce COMPASS (COdility’s Multi-dimensional Programming ASSessment), a benchmark that evaluates code generation in three critical dimensions: correctness, efficiency, and quality. COMPASS consists of 50 competitive programming problems from real coding competitions hosted by Codility, providing comprehensive percentile rankings of LLMs against human performance baselines from 393,150 submissions. We developed COMPASS to serve as a navigational aid for the field, enabling researchers and professionals to pinpoint where current models truly stand in terms of real-world performance, and to chart a clear path forward toward robust, efficient, and maintainable AI-driven code generation. II Related Work and Motivation II-A Limitations of Current Benchmarks Most existing benchmarks reduce evaluation to a single axis: functional correctness. HumanEval [ 1 ] helped popularize this approach, and later benchmarks such as APPS [ 7 ] , SWE - bench [ 3 ] , DS - 1000 [ 8 ] , and HackerRank - ASTRA [ 4 ] largely preserve this narrow focus, even when applied to more complex, real-world tasks. This leads to two critical blind spots: • Efficiency: A brute-force \mathcal{O}(n^{3}) implementation is treated as equivalent to an optimal \mathcal{O}(n\log n) algorithm as long as both pass test cases designed to assess only code functionality. This fails to reflect real-world software development, where efficiency determines feasibility and scalability. • Quality: Key aspects such as maintainability, readability, modularity, and adherence to best practices—critical for long-term productivity—are largely ignored in prominent evaluations. Current benchmarks prioritize syntactic correctness while overlooking principles of sustainable and scalable software engineering [ 5 , 6 ] . II-B The Need for Multi-Dimensional Assessment The importance of efficiency and quality is far from theoretical. Competitive programming platforms such as Codeforces and TopCoder impose strict runtime and memory constraints, while decades of software engineering research emphasize multi-dimensional metrics for assessing code quality. However, current code generation benchmarks fall short of capturing these essential dimensions. By relying primarily on pass@k metrics, they reduce software development to a binary pass/fail task—overlooking the efficiency, quality, and sustainability that define real engineering value. The result is a skewed portrayal of model capabilities, risking overestimation of their readiness for real-world use. This reveals a critical disconnect between what benchmarks measure and what modern software development truly demands. III COMPASS Benchmark Design COMPASS includes 50 coding challenges selected from Codility’s public contests held between 2011 and 2021. These problems span a wide range of algorithmic domains and difficulty levels, and are accompanied by large-scale human performance data, enabling more nuanced and multidimensional model evaluation. Participants could complete these challenges using any of 25 + mainstream programming languages, including C, C++, Python, Java, JavaScript, Go, Rust, Swift, Kotlin, PHP, Ruby, C#, TypeScript, and others. This diversity ensures that solutions reflect a broad range of programmer preferences and constraints. In this initial study, we restricted model-generated solutions to Python 3 to control for variance in performance outcomes across language implementations. Future research using the COMPASS benchmark will expand this analysis to include a broader set of programming languages. Each problem in COMPASS has been attempted by thousands of participants in real programming competitions, resulting in 393,150 human submissions. This large-scale dataset provides empirical insight into common patterns of failure and variation in problem-solving approaches under time-constrained conditions. Problems were selected to ensure diversity in topic coverage and to expose variation in reasoning depth, algorithmic efficiency, and implementation robustness—dimensions not captured by correctness-only benchmarks. To contextualize model performance, we report aggregate statistics from the full set of historical human submissions (see Appendix A ) along with percentile scores for each model against human-based norms. Overall benchmark data from human submissions are: • Average human mean score: 23.31% (range: 11.00%–40.40%). • Average human median score: 9.35% (range: 0.00%–44.00%). • Average human standard deviation: 31.08. • Average human skew: +1.36 . The positive average skew in human scores indicates a long tail of low-scoring attempts, evidence of high difficulty and common pitfalls such as inefficient solutions, misinterpreted constraints, or failure to handle edge cases. In short: these are hard problems, even for experienced programmers. By anchoring the benchmark in large-scale human data from open competitions, COMPASS offers a high-signal, multi-dimensional evaluation of code generation systems, assessing not just whether a model produces a working solution, but how it performs under the same pressure-tested conditions that reveal meaningful variation in human skill. Appendix A shows the human performance baselines on all 50 COMPASS problems. III-A Multi-Dimensional Evaluation Framework COMPASS evaluates each model solution across three independent dimensions: functional correctness , computational efficiency , and code quality . This structure reflects how software is evaluated in the real world: an examination of not just whether it works, but how well it works and how