Performance Evaluation of Large Language Models for High-Performance Code Generation: A Multi-Agent Approach (MARCO)

Asif Rahman, Veljko Cvetkovic, Kathleen Reece, Aidan Walters, Yasir Hassan, Aneesh Tummeti, Bryan Torres, Denise Cooney, Margaret Ellis, and Dimitrios S. Nikolopoulos · 2025

Read Performance Evaluation of Large Language Models for High-Performance Code Generation: A Multi-Agent Approach (MARCO) by Asif Rahman, Veljko Cvetkovic, Kathleen Reece, Aidan Walters, Yasir Hassan, Aneesh Tummeti, Bryan Torres, Denise Cooney, Margaret Ellis, and Dimitrios S. Nikolopoulos (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03906.txt.

Introduction

Rahman and co-authors speed up language-model code writing for high-performance computing, in a 2025 paper. They use a team of cooperating AI agents. Supercomputing code needs careful tuning for parallelism and memory that general models often miss. Their framework, MARCO, splits the work: one agent writes code and another grades its performance. A feedback loop refines the result.

A web-search agent fetches current optimization techniques from recent papers and proceedings, filling gaps in the model's training. On a standard problem set, the system cut run time by about fifteen percent against a strong model alone. Adding web search improved performance by about thirty-one percent over the base system.

Catalog record

CREATORAsif Rahman, Veljko Cvetkovic, Kathleen Reece, Aidan Walters, Yasir Hassan, Aneesh Tummeti, Bryan Torres, Denise Cooney, Margaret Ellis, and Dimitrios S. Nikolopoulos
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-03906

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Performance Evaluation of Large Language Models for High-Performance Code Generation: A Multi-Agent Approach (MARCO) Asif Rahman, Veljko Cvetkovic, Kathleen Reece, Aidan Walters, Yasir Hassan, Aneesh Tummeti, Bryan Torres, Denise Cooney, Margaret Ellis, and Dimitrios S. Nikolopoulos Abstract Large language models (LLMs) have transformed software development through code generation capabilities, yet their effectiveness for high-performance computing (HPC) remains limited. HPC code requires specialized optimizations for parallelism, memory efficiency, and architecture-specific considerations that general-purpose LLMs often overlook. We present MARCO (Multi-Agent Reactive Code Optimizer), a novel framework that enhances LLM-generated code for HPC through a specialized multi-agent architecture. MARCO employs separate agents for code generation and performance evaluation, connected by a feedback loop that progressively refines optimizations. A key innovation is MARCO’s web-search component that retrieves real-time optimization techniques from recent conference proceedings and research publications, bridging the knowledge gap in pre-trained LLMs. Our extensive evaluation on the LeetCode 75 problem set demonstrates that MARCO achieves a 14.6% average runtime reduction compared to Claude 3.5 Sonnet alone, while the integration of the web-search component yields a 30.9% performance improvement over the base MARCO system. These results highlight the potential of multi-agent systems to address the specialized requirements of high-performance code generation, offering a cost-effective alternative to domain-specific model fine-tuning. Emails: {asifr, veljko, katkreece501, aidanw, yasirh, artummeti, bryantorres, denise05, maellis1, dsn}@vt.edu I Introduction High-performance computing (HPC) represents the pinnacle of computational power, harnessing clusters of computing resources to transcend the limitations of individual machines. The fundamental advantage of HPC lies in its implementation of parallel processing techniques that maximize the performance of processor clusters, enabling complex data applications and mathematical calculations that would otherwise be infeasible [ 34 ] . HPC has been instrumental in driving innovation across diverse domains including climate modeling, astrophysics simulations, pharmaceutical research, energy optimization, financial risk analysis, and training state-of-the-art machine learning models, particularly Large Language Models [ 7 , 11 , 16 , 21 , 31 , 49 ] Large Language Models (LLMs) have revolutionized technology across multiple domains, democratizing access to high-level code through natural language interfaces. These models have evolved from conversational assistants to autonomous entities capable of executing tasks traditionally performed manually, such as code generation [ 20 , 45 , 29 , 37 , 26 ] . The ability of LLMs to generate, test, and evaluate code has significantly enhanced software engineering capabilities, reducing development cycles and increasing productivity. Despite their general utility, LLMs face significant challenges when applied to domain-specific knowledge requirements. According to the ParEval benchmark, both open-source and commercial LLMs demonstrate substantially lower accuracy in generating correct and efficient parallel code compared to serial code [ 34 ] . This limitation stems from LLMs’ training methodology, which typically involves diverse internet-sourced data rather than specialized domain knowledge. The relative scarcity of HPC code examples available online further exacerbates this knowledge gap [ 34 , 7 ] . LLMs have also been successfully deployed in parallel program verification [ 19 , 40 ] . However, the unique demands of high-performance computing—requiring both rigorously tested and optimized code for parallelism, memory efficiency, and execution speed—raise a critical research question: How can we effectively leverage LLMs to comprehensively address the HPC-specific programming challenges? A promising approach to addressing LLM limitations is the implementation of Artificial Intelligence (AI) Agents—autonomous LLM-based systems capable of performing complex tasks through customized workflows and specialized tools [ 48 , 1 , 27 , 15 , 41 ] . AI agents extend beyond natural language processing to encompass decision-making, problem-solving, and execution capabilities. By utilizing external tools, these agents can access up-to-date information and effectively complete domain-specific tasks. Multi-agent systems further enhance this capability by distributing complex workloads across specialized agents, enabling the handling of tasks that exceed the capabilities of any single agent. I-A Research Contributions This paper presents MARCO (Multi-Agent Reactive Code Optimizer), a novel framework that fundamentally reimagines code optimization for high-performance computing. Unlike traditional approaches that rely on monolithic LLMs or extensive fine-tuning, MARCO implements a specialized multi-agent architecture with several key innovations: • Specialized Agent Architecture: MARCO introduces a clear separation between code generation, performance evaluation and testing agents, allowing each to develop distinct expertise. The code generation agent specializes in implementing advanced optimization techniques including cache locality optimization, parallelization through OpenMP or CUDA, and vectorization strategies. Meanwhile, the evaluation agent focuses on rigorously assessing execution time, memory usage, and algorithmic complexity, while the testing agent continuously evaluates output correctness. • Adaptive Feedback Mechanism: Unlike traditional single-pass LLM code generation, MARCO implements an iterative feedback loop between its specialized agents. The evaluation agent collects comprehensive performance metrics that inform subsequent optimization attempts, creating a progressive refinement process that consistently improves code quality beyond what conventional approaches can achieve. • Real-time Knowledge Integration: MARCO incorporates a novel web-search component that actively retrieves the latest optimization techniques from research publications and conference proceedings, effectively bridging the knowledge gap between pre-trained LLM cutoff dates and current state-of-the-art HPC practices. This dynamic knowledge acquisition represents a significant advancement over static fine-tuning approaches. • Cost-Effective Implementation: By strategically separating optimization and evaluation processes, MARCO minimizes token usage and API call volume, offering a more economical alternative to expensive model fine-tuning or training specialized HPC models from scratch. To assess the effectiveness of our approach, we first assembled a thorough evaluation dataset, which included the LeetCode 75 problem set along with an additional 10 challenging problems. Next, we implemented MARCO and compared its performance to leading language models such as GPT-4o, Claude 3.5 Sonnet, DeepSeek Coder-V2, and Llama 3.1. Additionally, we conducted detailed ablation studies to measure the contribution of each component within the MARCO architecture. Our extensive experimental evaluation demonstrates that MARCO achieves a significant 14.6% reduction in average runtime on the LeetCode 75 problem set compared to Claude 3.5 Sonnet alone, while the integration of the web-search component yields a 30.9% performance improvement over the base MARCO system. These results establish MARCO as a transformative approach to HPC code optimization that outperforms current state-of-the-art models without requiring specialized training or fine-tuning. The remainder of this paper is organized as follows: Section II discusses background and related work, Section III details our methodology, Section IV presents experimental results, Section VI provides an in-depth analysis of the MARCO system, Section VII discusses our findings, and Section VIII concludes with future research directions. II Background & Related Work II-A LLMs for Code Generation LLMs are able to perform a wide range of tasks, such as code generation and understanding, through parameters and their transformer architecture [ 45 , 29 , 37 , 27 ] . LLMs have billions of parameters that enable the LLM to pick up on patterns that are passed into it during the training. The relationship between parameter size and model effectiveness is linear; with an increase in the number of parameters, the chance that the LLM can solve higher-complexity or data-intensive problems increases [ 23 , 16 , 13 ] . Alongside the high number of parameters is the transformer architecture, otherwise known as neural networks. These networks are intended to mimic the structure of a human brain’s neurons to predict intended outcomes. Chatbots like ChatGPT were the first to provide general-purpose assistance across numerous domains. LLMs

Read the complete text