OpenCompass: A Universal Evaluation Platform for Large Language Models
OpenCompass Team, Shanghai AI Laboratory · 2026
Read OpenCompass: A Universal Evaluation Platform for Large Language Models by OpenCompass Team, Shanghai AI Laboratory (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19276.txt.
Introduction
Maosong Cao and colleagues present OpenCompass, an open platform for judging large language models across many fields at once. Existing benchmark suites vary in task type and scoring, and their scattered workflows make large cross-domain comparisons slow. The platform couples a modular design with high concurrency, splitting work into configuration, task splitting, scheduling, execution, and display. It supports rule-based, judge-based, and staged scoring, helping teams spot a model's strengths and gaps before further tuning.
Catalog record
| CREATOR | OpenCompass Team, Shanghai AI Laboratory |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35613 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.19276
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OpenCompass: A Universal Evaluation Platform for Large Language Models OpenCompass Team, Shanghai AI Laboratory Abstract In recent years, the field of artificial intelligence has undergone a paradigm shift from task-specific small-scale models to general-purpose large language models (LLMs). With the rapid iteration of LLMs, objective, quantitative, and comprehensive evaluation of their capabilities has become a critical link in advancing technological development. Currently, the mainstream static benchmark dataset-based evaluation methods face challenges such as the diversity of task types, inconsistent evaluation criteria, and fragmentation of data and processing workflows, making it difficult to efficiently conduct cross-domain and large-scale model evaluation. To address the aforementioned issues, this paper proposes and open-sources OpenCompass, a one-stop, scalable, and high-concurrency-supported general-purpose LLM evaluation platform. Adhering to the design philosophy of modularization and component decoupling, the platform boasts three core advantages: high compatibility, flexibility, and high concurrency. The core architecture of OpenCompass comprises five key components: the Configuration System, Task Partitioning Module, Execution and Scheduling Module, Task Execution Unit, and Result Visualization Module. Its workflow provides rule-based, LLM-as-a-Judge, and cascaded evaluators to adapt to the requirements of different task scenarios. Supporting mainstream benchmark datasets across multiple domains, including knowledge, reasoning, computation, science, language, code, etc., the platform offers a unified and efficient LLM evaluation tool for both academia and industry, facilitating the accurate identification of strengths and weaknesses of LLMs as well as their subsequent optimization. † † * Code is at https://github.com/open-compass/opencompass 1 Introduction In recent years, the field of artificial intelligence has undergone a paradigm shift from task-specific small-scale models to general-purpose large language models (LLMs). Compared with traditional models that are small in parameter size and often require fine-tuning for specific downstream tasks, large language models exhibit remarkable generalization and emergent abilities through pre-training on massive corpora. This scale effect enables LLMs not only to tackle tasks on which they have not been explicitly trained, but also to possess powerful in-context learning capabilities. Although current large language models are rapidly evolving toward a multimodal orientation, integrating multiple perceptual modalities such as vision and audition, text processing capability — as the core carrier for logical reasoning, knowledge storage, and instruction interaction — remains a crucial cornerstone for the construction of high-level machine intelligence. With the rapid iteration of LLMs, the objective and quantitative evaluation of their capabilities has become a critical link in advancing technological development. Constructing a comprehensive capability evaluation system typically encompasses multiple dimensions, including general knowledge reserve, language understanding and generation, complex instruction following, logical reasoning, and professional domain-specific capabilities in vertical fields (e.g., medicine, law, coding, etc.). Through the evaluation of general and domain-specific capabilities based on the aforementioned multi-dimensional framework, researchers can accurately identify the strengths and weaknesses of models, thereby providing data support for subsequent model optimization. Currently, the academic and industrial communities primarily adopt static benchmark dataset-based methods for LLM evaluation. This approach quantifies the performance of models on specific tasks by examining its answer accuracy or generation quality across various standard test suites (e.g., MMLU [ 6 ] , HLE [ 13 ] , GPQA [ 15 ] .). However, a holistic assessment of the overall quality of LLMs often requires testing across dozens or even hundreds of benchmarks covering diverse domains and difficulty levels. Due to the diversity of task types and the variability of evaluation criteria, unified management of these scattered tests has become an extremely arduous task. Specifically, these differences mainly include: inconsistent evaluation methods for model outputs across benchmarks (e.g., regular expression matching, ROUGE scoring, model-based scoring), high sensitivity of results to prompt engineering, and highly fragmented preprocessing and postprocessing of data and model outputs among different benchmarks. To address the fragmentation and standardization challenges faced in the aforementioned evaluation process, we have developed and open-sourced OpenCompass — a one-stop, scalable, and high-concurrency-supported general-purpose LLM evaluation platform. It aims to provide a unified and efficient evaluation tool for both academia and industry. Specifically, OpenCompass exhibits the following prominent advantages: 1. Comprehensive and Extensive Support for Models and Datasets : OpenCompass supports the evaluation of 100+ mainstream datasets, covering dimensions including disciplinary knowledge, linguistic competence, factual knowledge, comprehension ability, reasoning capability, and safety compliance, as well as specialized capabilities such as long-text processing and code generation. Meanwhile, it is compatible with multiple types of large models, including HuggingFace-based models, accelerated inference engine models, API models, and various custom models. 2. Flexible and Diversified Evaluation Strategies : OpenCompass integrates multiple evaluation paradigms under subjective evaluation and objective evaluation. It also supports the customization of evaluation logic, data preprocessing, and postprocessing methods across different benchmarks. 3. Efficient Concurrent Evaluation Architecture : To address the challenges of high computational overhead and long response times in large model evaluation, OpenCompass is designed with an efficient task partitioning mechanism. It supports large-scale, cluster-based parallel inference, significantly reducing the time required to complete full-scale evaluations on massive datasets. 2 Preliminary LLM evaluation refers to the systematic assessment of large language models across key dimensions such as knowledge breadth, reasoning ability, generalization performance, and safety through a series of standardized testing methods. In existing evaluation systems, the evaluated objects are typically categorized into two types: Base Models, i.e., models pretrained solely on massive corpora with text continuation capabilities; and Chat Models, i.e., models that have undergone Supervised Fine-Tuning (SFT) or Reinforcement Learning from Human Feedback (RLHF), equipped with the abilities to follow complex instructions and conduct multi-turn interactions. For different types of models and tasks, evaluation methods are mainly divided into the following two categories: Objective Evaluation : Primarily targeted at tasks with definitive gold standard answers, such as multiple-choice questions, mathematical question answering, or code generation. This type of evaluation achieves efficient, low-cost, and fully reproducible automated assessment by comparing model outputs with gold standard answers, utilizing quantitative metrics including Accuracy, Exact Match, or F1 score. Subjective Evaluation : Mainly applied to open-ended generation tasks, such as creative writing, logical debate, or complex instruction response. Due to the lack of a unique gold standard answer, this type of evaluation typically relies on the LLM-as-a-Judge paradigm to perform multi-dimensional quality scoring on generated content across dimensions like coherence, usability, and instruction adherence. Typically, a complete evaluation task executed on static benchmarks follows a typical pipeline consisting of three phases: Data Preprocessing, Model Inference, and Result Evaluation. Data Preprocessing : Reads samples from raw datasets, performs format standardization and field alignment, and converts them into the input format required by the model. Model Inference : Constructs prompts and inputs them into the LLM for forward propagation to generate responses. Inference paradigms vary across different model types: base models generally adopt perplexity-based methods, while chat models commonly use generation-based approaches. Additionally, to address the efficiency challenges of large-scale evaluation, this phase typically involves batch inference and the scheduling of distributed computing resources. Result Evaluation : As the final link in the evaluation pipeline, it aims to convert the raw outputs into quantifiable scores. This phase involves post-processing of model outputs, answer extraction, and scoring. 3 Architecture Figure 1 : Overview of the OpenCompass architecture The project architecture of OpenCompass adheres to the design philosophy of modularization and component decoupling, aiming to construct a general-purpose evaluation framework with high efficiency, high compatibility, and