Benchmark Tracking System for Performance Monitoring
Billingsley, Braedon, Cottam, Joseph A. · 2025
Read Benchmark Tracking System for Performance Monitoring by Billingsley, Braedon, Cottam, Joseph A. (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/doe-osti-2584398-benchmark-tracking-system-for-performanc.txt.
Introduction
PNNL report 38155, by Braedon Billingsley and Joseph Cottam in August 2025, builds a memory for software benchmarks. The subject is Lamellar, an asynchronous runtime for high-performance computing developed at Pacific Northwest National Laboratory. Before this work, Lamellar's benchmark results were hard to track across code versions, so slow regressions could hide inside routine commits. The project set out to make performance history systematic and searchable.
Three pieces compose the system. Benchmark output was standardized into JSON Line records carrying execution time, hardware, and environment fields. Results are versioned in a dedicated git repository, chosen after weighing other data stores. A suite of Python tools enriches results with metadata and supports queries across commit history, hardware configurations, and benchmark variants. A cheap first-pass check trims the computing cost of comparisons. For a runtime whose whole purpose is speed, knowing exactly when it got slower is the point.
Catalog record
| CREATOR | Billingsley, Braedon, Cottam, Joseph A. |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| LICENSE | Public Domain |
| RIGHTS | PUBLIC DOMAIN |
| SOURCE | DOE OSTI |
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PNNL-38155 Benchmark Tracking System for Performance Monitoring August 2025 Braedon Billingsley Joseph Cottam Prepared for the U.S. Department of Energy under Contract DE-AC05-76RL01830 Choose an item. This report was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government nor any agency thereof, nor Battelle Memorial Institute, nor any of their employees, makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or any agency thereof, or Battelle Memorial Institute. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof. PACIFIC NORTHWEST NATIONAL LABORATORY operated by BATTELLE for the UNITED STATES DEPARTMENT OF ENERGY under Contract DE-AC05-76RL01830 Printed in the United States of America Available to DOE and DOE contractors from the Office of Scientific and Technical Information, P.O. Box 62, Oak Ridge, TN 37831-0062 www.osti.gov ph: (865) 576-8401 fox: (865) 576-5728 email: [email protected] Available to the public from the National Technical Information Service 5301 Shawnee Rd., Alexandria, VA 22312 ph: (800) 553-NTIS (6847) or (703) 605-6000 email: [email protected] Online ordering: http://www.ntis.gov Choose an item. PNNL-38155 Benchmark Tracking System for Performance Monitoring August 2025 Braedon Billingsley Joseph Cottam Prepared for the U.S. Department of Energy under Contract DE-AC05-76RL01830 Pacific Northwest National Laboratory Richland, Washington 99354 Benchmark Tracking System for Performance Monitoring Braedon Billingsley Abstract Benchmarking is essential for high-performance software development, particularly for monitoring performance across code iterations. This project focused on enhancing the benchmarking process for Lamellar, an asynchronous runtime for High-Performance Computing (HPC) systems developed at Pacific Northwest National Laboratory. Prior to this work, benchmark results were difficult to track and compare across code versions, presenting significant challenges in identifying performance regressions and long-term trends. The primary objective was to establish a systematic, reproducible approach for measuring performance and detecting regressions following code commits. Our methodology involved three key components: standardizing benchmark outputs, implementing data versioning, and developing analysis tools. We standardized the benchmark output format to JSON Line records containing specific fields (execution time, hardware specifications, and environmental variables). To address data management challenges, we evaluated several options and eventually chose a git repository dedicated to benchmark data. We developed a suite of Python tools that processed benchmark results, enriched them with metadata, and facilitated search in the repository. The resulting system enables more efficient filtering and comparison of performance metrics across commit histories, hardware configurations, and benchmark variants through a unified query interface. Our implementation reduces computational overhead by first checking for existing results through configuration matching before initiating new benchmark runs, thereby conserving resources. The system has been validated by Lamellar developers. It organizes results by benchmark type and build configurations for efficient retrieval. Future developments include a planned Large Language Model interface for predicting benchmark performance, incorporating the criterion package for statistical analysis, which will enable automated detection of statistically significant performance changes, and integration with continuous integration pipelines. Despite these enhancements being reserved for future work, this project has successfully provided the Lamellar development team with a framework for maintaining consistent performance standards and identifying optimization opportunities across workloads and hardware environments. Introduction Software performance optimization relies on consistent benchmarking to guide development decisions. In high-performance computing (HPC) environments, where applications run across distributed systems, even minor performance regressions can substantially impact computational efficiency. Lamellar is an HPC runtime built using Rust, aiming to offer a productive and safe development environment. It is a modern, performance-oriented runtime using Partitioned Global Address Space (PGAS) programming and asynchronous tasking program models [1]. While Lamellar has demonstrated promising results for distributed memory parallelism, its ongoing development faces a common challenge in HPC software engineering: maintaining consistent performance across an evolving codebase. This project focused on tracking performance across code revisions, hardware configurations, and varied workloads (as opposed to micro-benchmarks focused on hotspot identification). Several approaches exist for addressing this challenge, including continuous integration performance testing and automated regression detection systems, but implementing these for HPC environments presents unique difficulties. This project addresses needs within the Lamellar development workflow to systematically capture, store, and analyze benchmark results across its development lifecycle. By creating benchmarking infrastructure that connects performance metrics directly to code revisions, we enable data-driven optimization decisions and provide early detection of potential performance regressions. The remainder of this report details our technical approach to designing and implementing this benchmarking system, the specific contributions made during the internship period, and recommendations for future enhancements to further support the Lamellar development team. Progress Problem Analysis and Solution Design The initial phase of this project involved a review of Lamellar's existing benchmarking workflow, and the challenges faced by developers. Through discussions with the Lamellar development team, we identified requirements for an effective benchmarking system. Results needed to be directly tied to specific git commits in both the Lamellar and Lamellar Benchmarks repositories. The system needed to support comparison of results across different hardware configurations while including comprehensive metadata about the execution environment. Robust query capabilities were essential for filtering and analyzing results across multiple revisions, all while integrating smoothly with existing development workflows. After identifying these requirements, we evaluated several potential approaches. • Data Version Control (DVC) [2] • Git Large File Storage (LFS) [4] • Criterion [3] • LakeFS [5] • Dolt [6] • Git [7] DVC initially seemed promising for its ability to version large datasets alongside code. However, testing revealed that DVC introduced unnecessary complexity for our use case, particularly in how it managed data across repository branches. Git LFS, while suitable, was ruled out due to licensing costs. Criterion, though a major benchmarking framework in the Rust ecosystem, wasn't selected due to its limited support for benchmarking across multiple environments, which was a key requirement for our project. Criterion excels at statistical rigor for function-level microbenchmarks but was not designed for comparing results across different hardware configurations with environmental metadata. Other options like Dolt (versioned relational databases) and LakeFS (data lake management) were beyond our use case requirements. Based on this analysis, we decided to use a git repository for simplicity. We designed a solution with three core components: a standardized benchmark result format, a dedicated git repository for result storage, and a suite of Python tools for data processing and analysis. This approach balanced simplicity with the flexibility needed to support complex queries across benchmark results. Implementation of Benchmark Result Standardization The first technical contribution involved standardizing the output format for Lamellar benchmarks. We implemented a JSON Lines-based format where each benchmark execution generates a structured record containing performance metrics such as timing measurements, throughput rates, and data transfer statistics, along with benchmark-specific configuration parameters that define the test conditions. The format also captures comprehensive system information including hardware specifications, operating system details, and processor characteristics, as well as execution metadata such as run timestamps and dates for