CelloAI Benchmarks: Toward Repeatable Evaluation of AI Assistants
Mohammad Atif 1 , Kriti Chopra 1 , Fang-Ying Tsai 2 , Ozgur O. Kilic 1 Tianle Wang 1 , Zhihua Dong 1 , Douglas Benjamin 1 , Charles Leggett 3 , Meifeng Lin 1 , Paolo Calafiura 3 , Salman Habib 4 · 2026
Read CelloAI Benchmarks: Toward Repeatable Evaluation of AI Assistants by Mohammad Atif 1 , Kriti Chopra 1 , Fang-Ying Tsai 2 , Ozgur O. Kilic 1 Tianle Wang 1 , Zhihua Dong 1 , Douglas Benjamin 1 , Charles Leggett 3 , Meifeng Lin 1 , Paolo Calafiura 3 , Salman Habib 4 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-01051.txt.
Introduction
Atif and ten colleagues build CelloAI, a repeatable test suite for AI coding help in high-energy physics and high-performance computing. Existing tests ignore science limits and huge performance-critical codebases. Three tracks cover doc comments, GPU kernel code, and graphical data study with vision models. With repeatability, auto scoring, and domain-realistic failure modes, the suite lets teams compare models fairly and improve reliability for science software.
Catalog record
| CREATOR | Mohammad Atif 1 , Kriti Chopra 1 , Fang-Ying Tsai 2 , Ozgur O. Kilic 1 Tianle Wang 1 , Zhihua Dong 1 , Douglas Benjamin 1 , Charles Leggett 3 , Meifeng Lin 1 , Paolo Calafiura 3 , Salman Habib 4 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27751 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.01051
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CelloAI Benchmarks: Toward Repeatable Evaluation of AI Assistants Thanks: [email protected] Mohammad Atif 1 , Kriti Chopra 1 , Fang-Ying Tsai 2 , Ozgur O. Kilic 1 Tianle Wang 1 , Zhihua Dong 1 , Douglas Benjamin 1 , Charles Leggett 3 , Meifeng Lin 1 , Paolo Calafiura 3 , Salman Habib 4 Abstract Large Language Models (LLMs) are increasingly used for software development, yet existing benchmarks for LLM-based coding assistance do not reflect the constraints of High Energy Physics (HEP) and High Performance Computing (HPC) software. Code correctness must respect science constraints and changes must integrate into large, performance-critical codebases with complex dependencies and build systems. This paper’s primary contribution is the development of practical, repeatable benchmarks that quantify LLM performance on HEP/HPC-relevant tasks. We introduce three evaluation tracks: code documentation benchmarks measure an LLM’s ability to generate Doxygen-style comments, code generation benchmarks evaluate end-to-end usability on representative GPU kernels, and graphical data analysis benchmarks evaluate vision-enabled LLMs. These benchmarks provide a unified framework for measuring progress in scientific coding assistance across documentation quality, code generation robustness, and multimodal validation analysis. By emphasizing repeatability, automated scoring, and domain-relevant failure modes, the suite enables fair comparisons of models and settings while supporting future work on methods that improve reliability for HEP/HPC software development. I Introduction Recent advances in large language models (LLMs) have enabled new tools that can assist scientists with two pain points in research software development cycle: (i) producing clear natural-language documentation for complex code, and (ii) generating or modifying code to accelerate development [ 6 , 13 , 14 ] . Yet, in high-performance computing (HPC) environments, practical adoption remains constrained by the difficulty of providing sufficient context for massive, sparsely documented codebases. Moreover, heterogeneous architectures amplify these challenges, where mistakes in translation, data movement, or dependency handling can silently invalidate results or degrade performance. CelloAI address these constraints as a locally hosted, retrieval-augmented coding assistant purpose-built for scientific HPC workflows [ 2 ] . CelloAI’s goal is to make LLM assistance reliable for large scientific repositories by assembling high-quality contexts before generation. The system is designed around three core ideas: (1) retrieval-augmented generation that pulls from both scientific text and source code; (2) syntax-aware code chunking to preserve semantic boundaries to avoid fragmented retrieval; and (3) callgraph-aware prompt enhancement to provide dependency context (caller/callee), so LLM responses respect the execution flow of real applications. Together, these mechanisms support documentation, code understanding, and code generation tasks while maintaining transparency and safety requirements that are essential in scientific computing. This paper expands CelloAI into a benchmark-driven framework spanning three HEP-relevant evaluation tracks with methodologies applicable to other HPC domains. Benchmarking is essential because qualitative, anecdotal impressions do not provide a repeatable basis for comparing models or strategies across large codebases. Standardized tasks and automated scoring make performance claims testable, enable fair comparisons across systems, and reveal failure modes that are easy to miss in smaller studies. The main contribution of this paper is to provide practical strategies for developing benchmarks for code documentation, code generation/porting, and graphical data analysis, complementing existing CelloAI feaures. These benchmarks also provide a foundation for measuring progress in AI technologies and enabling fair comparisons between different models. Existing benchmarks are strong for general coding competence and unit test-driven software fixes [ 9 , 8 , 17 ] , but they fall short of HEP/HPC needs where correctness requires scientific validation and performance [ 5 ] . Hence, we introduce benchmarks for code documentation , targeting structured documentation such as Doxygen-style comments, where success requires both complete coverage of parameters/returns and scientifically grounded descriptions. Second, we introduce benchmarks for code generation and transformation in HPC settings, emphasizing GPU porting where correctness, dependency awareness, and guardrailed behavior matter as much as raw generation ability. Third, we also extend CelloAI toward graphical data analysis benchmarks, leveraging histograms generated during HEP experiments and simulations. Our aim is to provide a unified and reproducible way to measure whether LLMs can generate code accurately, completely, consistently, and with scientific grounding under real HPC constraints. II Code Documentation Code documentation is critical in HEP and HPC software because these code-bases are characteristically large and long-lived, developed by distributed teams, and must maintain understandability and reproducibility despite decades of evolution with changing personnel and hardware. LLMs offer a practical way to automate parts of documentation by generating structured comments directly from function signatures and local code context, reducing the manual effort required to keep comments synchronized with rapidly changing implementations. However, scientific documentation requires domain-specific terminology, making output quality dependent on whether models can accurately infer scientific intent from limited context. Additionally, output quality is also sensitive to instruction adherence, since formats such as Doxygen require consistent structure alongside meaningful descriptions. Our objective is therefore to determine which LLMs (and their associated parameters) most reliably produce high-quality Doxygen-style comments for scientific/HPC functions. A central challenge is that assessing descriptive text’s correctness is inherently difficult. Hence, we formalize benchmarks by combining structural measures of completeness with semantic measures that probe whether the generated text is coherent and aligned with expert intent. II-A Doxygen-style Comment Generation with CelloAI CelloAI generates Doxygen-style comments by extracting code units and interface information, including signatures, argument names/types, and return types. It retrieves supporting context from the repository and linked scientific artifacts that describe the implemented algorithm or physics behavior. Using this context, the model produces syntactically valid Doxygen blocks with required tags (e.g., @brief , @param , @return ) and concise descriptions. II-B CelloAI-Doc-Bench CelloAI-Doc-Bench is a lightweight benchmark suite to evaluate how well LLMs generate structured documentation for scientific and high-performance computing code. It computes two core metrics: a coverage score, measuring the fraction of documented parameters and return tags relative to the ground truth, and a semantic similarity score, measuring the coherency of generated text and and alignment with expert-written comments. Together these metrics capture both completeness and contextual grounding of the generated documentation. II-B 1 Coverage Score To evaluate structural correctness, we extend the standard F1 score [cite] to a tag-level coverage score in CelloAI-Doc-Bench. The benchmark emphasizes coverage by ensuring that every parameter and return value in the function is reflected in @param and @return tags, and hence evaluates this with tag-level precision/recall and an F_{1} summary. We count the number of true positive (TP), false positive (FP), and false negative (FN) tags. In this context, a TP occurs when the LLM correctly includes a tag for an existing parameter or return value; an FP occurs when it adds a tag that does not exist in the ground truth; and a FN occurs when it omits a tag that should be present. The final F_{1} score is the harmonic mean of precision and recall scores defined as \displaystyle\begin{split}F_{1}&=\frac{2\times\text{Precision}\times\text{Recall}}{\text{Precision}+\text{Recall}},\\ \text{Precision}&=\frac{TP}{TP+FP},\text{Recall}=\frac{TP}{TP+FN}.\end{split} (1) Higher recall here means more complete documentation, while higher precision indicates fewer redundant tags, thus yielding an interpretable measure of documentation quality. II-B 2 Semantic Similarity While tag-based coverage score evaluates structural accuracy, it does not capture whether generated documentation is meaningful. To address this, we introduce two semantic similarity scores using cosine similarity between vector representations of descriptive text: • Differential Similarity:Coherent LLM documentation should show consistent parameter descriptions across related functions.