Annual Report CY2025 i STEP CY2025 Report v1.1 Table of Contents Table of Contents 1. BACKGROUND 1.1 Introduction This report gives an account of the Software Tools Ecosystem Project (STEP) during calendar year 2025 [1]. STEP seeks to ensure the stewardship and continued advancement of tools and supporting software for monitoring, analysis, and diagnosis of performance and behavior. Performance tools must proactively adapt to the challenges presented by emerging platform technologies such as new accelerators, storage devices, network technologies, and smart devices. Further challenges arise from changes to software stacks (e.g., vendor GPU monitoring libraries) and emerging application use cases, including advanced machine learning and workflow frameworks. STEP enables these tools to continue meeting the needs of application scientists and to provide a strong foundation for future Advanced Scientific Computing Research activities. STEP is a component of the Next Generation Scientific Software Technologies (NGSST) program and the Consortium for the Advancement of Scientific Software (CASS). The leadership team consists of three funded roles: • STEP Director: Terry Jones, Oak Ridge National Laboratory • STEP Deputy Director: Mike Jantz, University of Tennessee (since FY2026) 1 • STEP Events & Workforce Expertise: Suzanne Parete-Koon, Oak Ridge National Laboratory 1.2 Overall Project Goals and Opportunities STEP is a purpose-driven activity focused on achieving the following project goals: • Deliver software tools that achieve greater effectiveness and efficiency for applications running on a spectrum of computing systems within the DOE complex – from laptops to leadership class machines • Grow workforce expertise and shared experience • Gain leverage with partners who have similar requirements • Serve as a nexus of communication and information for tools stakeholders We seek to be an agile project that can work effectively across a broad range of funding levels. At present, we are pursuing strategies that accommodate very limited resources. We are prioritizing essential tasks, utilizing creative solutions, and maintaining clear communication with stakeholders to ensure we're delivering the most impactful results. If given the opportunity, our long term vision is to grow the scope of STEP by fostering engagement between community stakeholders and vendors to codesign hardware monitoring interfaces and software that affect tools, with the aim of enhancing efficient utilization of low-level performance monitoring and analysis capabilities. Moreover, we seek to support work through ongoing collaborative efforts by cross- cutting teams to address shared concerns across multiple tools and enhance long-term sustainability. We wish to support incubation of new technology as well as agile short-duration activities that have high potential to address rapidly evolving needs and opportunities. Finally, to the extent possible, we aim to support community-wide stewardship and advancement activities with broad impact on the tools ecosystem, e.g. standards, with the aim of enhancing performance portability and enabling efficient use of diverse hardware platforms. 1 Philip Carns (Argonne National Laboratory) served as the STEP Deputy Director before he left ANL in 2025. 1 STEP CY2025 Report v1.1 Narrative 1.3 Software Package Portfolio STEP includes the following five software packages: • HPCToolkit ...Software package lead: John Mellor-Crummey, Rice University John’s webpage • PAPI .............Software package lead: Heike Jagode, University of Tennessee Heike’s webpage • Dyninst ....... Software package lead: Bart Miller, University of Wisconsin Bart’s webpage • TAU ..............Software package lead: Sameer Shende, University of Oregon Sameer’s webpage • Darshan .........Software package lead: Orçun Yildiz, Argonne National Laboratory Orçun’s webpage Each software package has detailed accomplishments and future opportunities included later in this report. 1.4 Organization of Report The report is organized as follows. We follow this background section with a section that provides a synopsis for the activities of each STEP software package during calendar year 2025. We then step back and provide a holistic view of how the project gains synergy from combining efforts and activities; this section also discusses challenges faced by performance software tools. We then conclude the report and provide two appendices with additional information. 2. Software Package Status 2.1 HPCToolkit HPCToolkit is an integrated suite of tools for measurement and analysis of application performance on computers ranging from multicore desktop systems to GPU-accelerated supercomputers running applications at exascale. On CPUs, HPCToolkit profiles and traces applications by using statistical sampling of timers and hardware performance counters. On GPUs, HPCToolkit leverages vendor software frameworks for monitoring and analysis of GPU operations and instruction-level monitoring within GPU kernels. Context HPCToolkit is dependent upon vendor provided interfaces for measuring code performance on GPUs. Over the last 12 months, the HPCToolkit project team has engaged with both AMD and Intel about bugs and missing capabilities in their measurement interfaces. The project team provided extensive feedback to AMD about pre-releases of their emerging ROCprofiler- SDK interface, which supports profiling and tracing of GPU operations, time-based PC sampling within GPU kernels, and support for using hardware performance counters to measure events as kernels execute. AMD improved ROCprofiler-SDK based on feedback from Rice, released version 1.0.0 in March 2025, and version 1.1.0 in November 2025. The project team engaged Intel about their emerging PTI-GPU measurement substrate for profiling and tracing GPU operations and provided feedback that PTI-GPU’s initial support for synchronous callbacks when GPU operations are launched could not be used to supply correlation IDs needed to attribute costs of GPU operations to their invocations. In the fall of 2025, Intel added this missing capability. • Work with AMD to finalize ROCprofiler-SDK support for MI300. Implement HPCToolkit support for PC sampling on MI300, including collection and attribution of information about instruction stalls. AMD updated support for PC sampling in their March and November releases of Rocprofiler-sdk. The 2025.1.0 release of HPCToolkit in December 2025 included new support for both software and hardware (stochastic) PC sampling. This is a significant milestone as it began with convincing AMD 2 STEP CY2025 Report v1.1 Narrative to add hardware support for PC sampling into their GPUs in 2019 and working with them over the years since on their tools SDK that provides access to this capability. • Work with Intel to resolve showstoppers that present post-mortem analysis of instruction-level measurements on Aurora. Rice has continued a dialog with Intel about the fact that their tool support in Level Zero supplies fully- linked GPU executables back to tools and that these executables often differ across MPI ranks. These differences occur because GPU executables are fully linked by each MPI rank and linking by a rank incorporates addresses that depend upon the state of its memory allocator. Intel has neither fixed the problem nor provided a suitable workaround. As a result, it is only practical to use HPCToolkit to perform PC sampling on small-scale executions on Aurora. • Deploy new HPCToolkit capabilities for instruction-level performance monitoring at scale on Aurora, Frontier, and El Capitan. The 2025.1.0 release of HPCToolkit includes new support for hardware PC sampling on the Intel Data Center GPU Max Series GPUs in Aurora, software support for PC sampling in the AMD MI250X GPUs in Frontier, and both hardware and software support for PC sampling in AMD MI300A GPUs in El Capitan. • Improve HPCviewer user interface to better support analysis of large-scale performance data. Performance analysis of HPCToolkit’s HPCviewer user interface identified that when analyzing performance data for complex applications, filtering of Calling Context Tree (CCT) nodes (e.g. to hide unwanted details of measurements within implementations of MPI primitives) was very sluggish. Replacing an O(n²) algorithm for matching CCT nodes during metric filtering with an O(n) algorithm, reduced the time required to apply filters on large calling context trees by as much as 10x. • Mentor developers in hackathons and trainings at DOE laboratories. The HPCToolkit project team mentored participants in the 2025 El Capitan hackathon at Lawrence Livermore National Laboratory in January 2025, the Argonne Leadership Computing Facility INCITE GPU Hackathon inMay 2025, the Argonne Training Program on Extreme-Scale Computing in August, 2025, and the Argonne Leadership Computing Facility Hands-on High Performance Computing workshop in October, 2025. (Participation and travel for these hackathons were funded by LLNL and ANL and not STEP.) Property State Goal: Improve support for performance analysis at large scales Accomplishments: Completed a first full implementation of hpcanalysis – a new Impact Python-based API for programmatic analysis of large-scale performance data. hpcanalysis enables analysis tasks to be automated, integrated, and saved in Jupyter notebooks. Details of this work were published in an ICS 2025 paper “Analyzing the Performance of Applications at Exascale” (https://doi.org/10.1145/3721145.3730417) Goal: Improve sustainability of software for performance measurement on GPU- accelerated supercomputers. Sustainability Accomplishments: Replaced support for performance measurement on AMD GPUs based on AMD’s now deprecated Rocprofiler and Roctracer with support for performance measurement based on AMD’s new Rocprofiler-SDK, released in 2025. 3 STEP CY2025 Report v1.1 Narrative Property State Goal: Improve the quality and usability of HPCToolkit software Accomplishments: Over the last year, HPCToolkit’s hpcviewer user interface was improved in several ways. First, to simplify installation of hpcviewer, hpcviewer now Quality includes a builtin compatible Java runtime environment. Second, the project team improved the support for using hpcviewer on a desktop or laptop to access remote performance data by adding a capability for loading remote performance data in bulk rather than item-by-item. Third, the project team improved error reporting when establishing an ssh connection to a remote system. Finally, the team improved the performance of filtering calling contexts in profile and trace views. Test Smoke Durability Website Documentation Repository Spack E4S Suite Test. HPCToolkit Documentation Repository ✅ Spack ✅ 🚫 Support for measurement using hardware and Plan Build Release Operate software PC sampling on AMD GPUs Test Monitor Code Deploy Support for measurement using hardware PC Plan Build Release Operate sampling on Intel GPUs Test Deploy Code Monitor Updates to hpcviewer and hpcserver to better Plan Build Release Operate support remote performance analysis of large- Test Deploy scale executions Code Monitor Graduate research assistant Yumeng Liu defended her Ph.D. dissertation “Measurement and Analysis of GPU-accelerated Applications at Exascale” in September 2025. Graduate research assistant Yuning Xia, who has been working on PC sampling for Intel GPUs supported by a Rice University graduate fellowship, will be supported in part by the project as of January 2026. The STEP project provides partial support for development of HPCToolkit, Dyninst, PAPI, Tau, and Darshan. Common funding facilitates coordinated development. The HPCToolkit team continued work with members of the Dyninst team to analyze and remediate root causes of serialization when analyzing the machine code for Aurora’s default MPI library. The HPCToolkit team provided code to the Tau project team to add support for PC sampling. 4 STEP CY2025 Report v1.1 Narrative The HPCToolkit team has continued to work with HPSF on community-wide support for continuous integration. As a tool for analyzing and improving the performance of scientific software, HPCToolkit is well aligned with the goals of the Consortium for Advancement of Scientific Software (CASS). • Improve performance of HPCToolkit’s hpcstruct tool on complex application binaries (Q1). • Improve the usability of HPCToolkit for detailed performance analysis of GPU kernels using PC sampling (Q2) • Document and deploy the hpcanalysis software for programmatic GPU-accelerated analysis of performance data (Q4) 2.2 PAPI PAPI (Performance API) is a portable software library that provides a consistent interface for accessing hardware performance counters across diverse computing architectures. It addresses the long-standing challenge of vendor-specific and platform-dependent performance interfaces by enabling application developers and performance tools to collect meaningful performance data in a portable and reusable manner. Over time, PAPI has evolved beyond CPU counters to support GPUs, power and energy measurements, network interconnects, I/O activity, and software-defined events. By abstracting hardware heterogeneity, PAPI enables sustainable and portable performance monitoring across generations of HPC systems. Context PAPI serves as foundational middleware infrastructure for performance analysis in DOE HPC environments. It is widely used indirectly through performance tools such as TAU, Score-P, HPCToolkit, Caliper, CrayPat, and Vampir, and has been deployed for decades on leadership-class DOE systems. It enables: • Portable performance and energy measurement across architectures • Scientific application scalability and optimization • Architecture evaluation and co-design efforts • Performance portability across CPU and GPU generations Its role as enabling infrastructure allows scientific teams to focus on domain science while relying on a stable, vendor-neutral performance interface. The primary objectives for CY2025 focused on expanding architecture support, improving usability, and enhancing sustainability of the PAPI ecosystem. Key activities and accomplishments include: • Major Releases: o PAPI 7.2.0b1 (Aug 2024) o PAPI 7.2.0b2 (Mar 2025) o PAPI 7.2.0 (Jun 2025) • Expanded Hardware Support: o Full AMD GPU support via ROCprofiler-SDK (including sampling and multi-GPU) o Support for AMD Zen5 and latest Intel CPUs (Sapphire Rapids, Granite Rapids, etc.) o Support for heterogeneous CPU architectures (P-cores and E-cores) o Early ARM Neoverse V2 and RISC-V support 5 STEP CY2025 Report v1.1 Narrative • Usability Improvements: o Introduction of “topdown” metrics for actionable performance analysis o GPU preset metrics for portability o Reduction of NVIDIA event complexity via statistics qualifiers • Adoption and Robustness: o Improved interoperability across GPU software stacks (ROCm, CUDA) o Support for mixed GPU generations o Enhanced memory safety and robustness for production workloads These efforts lowered the barrier to adoption and improved portability across modern heterogeneous systems. Property State Goal: PAPI integrates new capabilities for emerging and advanced technologies, ensuring it is well-positioned to address the evolving needs of the HPC community and potentially expand its user base. Accomplishments: • PAPI now includes AMD’s ROCprofiler-SDK support for MI300/350 Impact • Improved handling of heterogeneous NVIDIA GPU systems (e.g., mixed compute capabilities) • New event qualifiers like “device” and “stats” to simplify and shorten event listings • Expanded RAPL support for AMD Family 25 (19h) • Intel CPUs such as Raptor Lake, Emerald Rapids, and Comet Lake S/H. • Integrated into widely used performance tools (TAU, Score-P, Caliper, etc.) • Over 1,740 citations for foundational publications; 242+ new citations (2024– 2025) • 4,000+ unique downloads for recent releases Goal: Ensure long-term maintainability, portability, and adaptability of the software. Accomplishments: Sustainability • Removed obsolete dependencies (e.g., perfctr, libpfm-3), reducing technical debt • Modular architecture enabling easy addition of new components and platforms • Component-aware testing infrastructure • Active GitHub-based development workflow (PRs, reviews) • Integration into Spack and E4S for reproducibility and deployment Goal: Improve software reliability, usability, and robustness. Accomplishments: Quality • Expanded automated CI testing across components (including GPUs and sensors) • Peer-reviewed development workflow • Improved memory safety and robustness for long-running jobs • Enhanced usability via presets and topdown metrics • Packaging and validation through Spack and E4S 6 STEP CY2025 Report v1.1 Narrative Test Smoke Durability Website Documentation Repository Spack E4S Suite Test. PAPI Documentation Repository ✅ Spack ✅ ✅ Finalized development and released new “rocp_sdk” PAPI component to support AMD Plan Build Release Operate GPUs/APUs (e.g., MI300/350) via ROCprofiler- Code Test Deploy Monitor SDK. New developed and released a new “topdown” PAPI component to support Intel’s topdown Plan Build Release Operate metrics and convert them into user-friendly Code Test Deploy Monitor percentage events. Finalized development and release of new statistics qualifiers for native CUDA events, to Plan Build Release Operate simplify access to 200K+ metrics on new NVIDIA Code Test Deploy Monitor GPUs. Finalized development and release of new PAPI Plan Build Release Operate Presets for GPUs, starting with CUDA-related PAPI Presets. Code Test Deploy Monitor PAPI development is carried out by a small, highly specialized team with deep expertise in hardware performance analysis and HPC systems. Strengths: • Strong domain expertise in performance analysis and architecture support • Long-term continuity and institutional knowledge Challenges: • Limited staffing constrains feature development and user support • Increasing complexity of heterogeneous systems (CPU + GPU + accelerators) requires broader expertise • Growing demand for AI-related performance metrics Needs: • Additional personnel for development, testing, and user support • Training in emerging architectures (AI accelerators, TPUs) • Continued support for sustainable software engineering practices PAPI is strongly aligned with the STEP and broader CASS ecosystem as a foundational performance infrastructure component. • Enables interoperability across multiple STEP software packages • Widely used by performance tools within the ecosystem • Supports DOE mission through performance portability and system evaluation 7 STEP CY2025 Report v1.1 Narrative Strengths of the ecosystem: • Collaboration across tools and facilities • Shared infrastructure (e.g., Spack, E4S) Opportunities for improvement: • Stronger cross-project integration around performance data • Increased coordination on AI performance analysis capabilities 1. Development and integration of new PAPI Presets for AMD GPUs. The goal of “PAPI Presets for GPUs” is to organize this large number of raw performance counter events into a more user- friendly and functional format. 2. Development and integration of new Presets for NVIDIA GPU components, with focus on mixed and half precision floating-point operations presets. 3. Finalize integration of PAPI's new ROCprofiler-SDK component into Spack and E4S, including "smoke tests" to quickly verify the basic functionality of all newly added and released PAPI features. 2.3 Dyninst Dyninst (and its related toolkits) is a binary code and instrumentation facility. It provides the ability to analyze binary code (both control and dataflow analyses) instrument the code efficiently. Dyninst also allows for modification of the executable. It supports such use cases as performance profiling, debugging, testing, tracing, security auditing and hardening, and architectural simulation. Its primary CPU platforms are x86, ARM, and Power, with recent new support for RISC-V. Dyninst also provides code analysis for AMDGPU, with instrumentation for that platform in progress. Currently, Dyninst provides binary analysis support for HPCToolkit and (in some modes) analysis and instrumentation for TAU. It is also the foundation for the LLNL STAT scalable debugger (in production and demonstrated effective at 5 million cores). Cray/HPE tools, including the ATP (abnormal termination processing debugger) and differential debugger are based on our toolkits. In addition, the Red Hat SystemTap monitoring tool is based on Dyninst, as is the new AMD Omnitrace. Currently in flight is support for the AMD GPUs allowing all current Dyninst tools to move to this platform. Context The Dyninst project is focused on binary code analysis and instrumentation for CPU and GPU systems. CPU code analysis and instrumentation is well established, with the main goal of adapting to new CPU designs as they are released, especially those associated with the leadership class systems. GPU analysis and instrumentation for the AMD platforms is underway. The Dyninst project continues to work closely with AMD, but response time from AMD continues to be a challenge. Our effort with them to get a machine-readable instruction semantics definition for AMDGPU has the challenge of long delays within AMD. We have worked with them to define a format and give immediate feedback on intermediate versions, but their response time is quite slow. We are also working with AMD on basic features needed for tool support on their GPUs. Currently, this support is quite primitive. We continue to work with them to understand the undocumented (or limited documented) features that we need to support Dyninst. As for the other STEP projects, the Dyninst project continues to operate at a funding level below what is needed to both innovate and support the new technologies and hardware that continue to emerge and be deployed. • Finalized instruction machine-readable instruction syntax specification with AMD. We are now in a mode where we update Dyninst as new definitions are released. 8 STEP CY2025 Report v1.1 Narrative • Continue to work with AMD to define their machine-readable instruction semantics definition. We are waiting now for AMD on this item. • Basic binary code analysis for AMD GPU (through MI300A) works. The analysis is sufficient to support basic control flow analysis for explicit control flow operations and dataflow analysis on the scalar and whole-vector level. • Control flow analysis for SIMD primitives based on execution masks is in progress. This analysis abstracts control flow structures that use execution masks (i.e., do not use explicit branching) into conventional CFG structures. Such structures should allow tools and users to operate on SIMD code in the same way that they operate on conventional code. • Instrumentation for SIMD code is in progress. We have demonstrated the ability to instrument at multiple levels of granularity, from whole kernel to individual lanes. Locking is used at the whole kernel level and avoided at the wavefront/warp and lane levels due to the use of separate instrumentation variables. Lane level instrumentation is facilitated by using vector instructions to instrument vector code (an innovation). Property State Goal: Continue to support the variety of tools that depend on Dyninst and support the leadership class systems. Accomplishments: Dyninst supports both HPCToolkit and TAU inside the STEP Impact project. It also supports the LLNL STAT scalable debugger tool (with victories debugging at millions of cores); Cray/HPE tools, including their ATP (abnormal termination processing debugger) and differential debugger; Red Hat’s SystemTap monitoring tool, and AMD’s OmniTrace tool. Many other projects use Dyninst for a variety of tool uses. The in-progress AMDGPU analysis and instrumentation will provide similar (and first in kind) support on that GPU platform. Goal: Move to LLVM-associated Capstone instruction decoder to help support parsing of evolving instruction sets. Sustainability Accomplishments: Prototyped this use on the RISC-V port of Dyninst (under separate funding). Currently do not have resources to apply this to other platforms. Instructions: Fill in the following table with links or checkboxes for each of the following software quality characteristics for your software. Test Smoke Durability Website Documentation Repository Spack E4S Suite Test. Dyninst Documentation Repository ✅ Spack ✅ 🚫 9 STEP CY2025 Report v1.1 Narrative Instructions: List any specific software features implemented in CY2024 that you would like to highlight (these do not necessarily map 1-1 with your impact sustainability and quality objectives). Mark checkboxes for each to indicate what steps have been completed in the software quality lifecycle. Dyninst CPU-based code analysis and Plan Build Release Operate instrumentation Code Test Deploy Monitor Basic code analysis for AMD GPU Plan Build Release Operate Code Test Deploy Monitor Control flow analysis for AMD GPU SIMD Plan Build Release Operate constructs Code Test Deploy Monitor Data flow analysis for AMD GPU SIMD Plan Build Release Operate constructs Code Test Deploy Monitor The primary support for STEP is a fraction of the time for research staff member James Kupsch. Kupsch is well experienced in the area. We continue to support his development needs with access to our technical partners and training activities. We look forward to a funding level that will again enable support graduate research assistants and develop their skills for the workforce. We interact closely with the STEP team and directly support the tools efforts of HPCToolkit and TAU. Through those projects (and other tools by AMD, Cray/HPE, LLNL, and Red Hat), we interact with the broader ecosystem. • Developing an instrumentation memory model for SIMD code. Such a model has to support instrumentation data allocations on a per-kernel-launch, per wavefront/warp, and per lane basis. • Developing a mechanism for linking instrumentation libraries into a GPU executable. • Demonstrating the SIMD control flow analysis, able to hoist code based on execution masks to conventional CFG structures. • GPU instrumentation working at the whole kernel, wavefront/warp, and lane granularities. 2.4 TAU TAU is performance evaluation tools that supports parallel profiling and tracing on HPC systems. It requires no changes to the application source code, build system, or binary. It also supports instrumentation using a plugin for the LLVM compiler, source instrumentation using the Program Database Toolkit (PDT), and binary rewriting of application binaries and DSOs using DyninstAPI. It supports automatic instrumentation for parallel programs written in C++, C, Fortran, Python, Julia, Java, Chapel, and UPC. Instrumentation can also be inserted using third party timing libraries such as GPTL, CAMTIMERS, PETSc, Perfstubs, Caliper, PhiProf, ROCTx, NVTx, and Score-P are supported. It supports instrumentation of OpenMP using OMPT including target offload directives, ROCm using ROCprofiler- SDK, CUDA using CUPTI, Kokkos, OpenCL, Intel Level Zero, and Python. TAU interfaces with PAPI to 10 STEP CY2025 Report v1.1 Narrative access hardware performance counter data and supports direct instrumentation as well as event-based sampling. TAU can generate traces in the native binary Perfetto trace format for timeline visualization of flamegraphs from each rank at Perfetto.dev. Context TAU is a widely used performance evaluation tool that has evolved over time to support most profiling and tracing packages on DOE computing systems at ALCF, OLCF, NERSC, Sandia, LLNL, and LANL. TAU supports instrumentation of application binaries at the routine level. Support for advanced instrumentation and Continuous Integration/Continuous Deployment (CI/CD) is important for sustaining TAU. Support for GPUs from Intel, AMD, and NVIDIA for CI/CD is key to sustain the TAU project. To advance the capabilities of dynamic instrumentation in TAU using DyninstAPI for targeted instrumentation of dynamic shared objects (DSOs) using a selective instrumentation file to specify the instrumentation regions of interest and initial support for CI/CD for GPU runtimes with MPI are CY2025 activities. Property State Goal: Support for selective instrumentation in rewriting DSOs with DyninstAPI at the routine level. Accomplishments: TAU now supports rewriting of DSOs using tau_run, a tool that is developed Impact using the DyninstAPI package with support for selective instrumentation. This is important for instrumentation of third-party libraries like HDF5 that are linked with an un-modified application binary. It helps generate performance data for the library without any modification to the source code, build system, or executable and targets specific code regions instead of the entire library for instrumentation. This feature has been updated and has been released in TAU v2.35 in November 2025. Goal: To improve TAU’s support for GPUs. Sustainability Accomplishments: TAU’s support for Intel, AMD, and NVIDIA GPUs has been updated. New versions of TAU (v2.34, v2.34.1, v2.35) have been released in CY25. Goal: To improve TAU’s support for CI/CD with MPI and GPU runtimes on Intel, AMD, and NVIDIA GPUs. Quality Accomplishments: TAU uses Frank at the University of Oregon for CI/CD. In CY25, support for Intel, AMD, and NVIDIA GPUs was added with tests run on 13 GPU architectures daily. In each of these GPU architecture, tests have been added to exercise the MPI instrumentation as well as cuda, or ROCprofiler-SDK, or level zero. While the initial integration is ready and being run in CI pipelines every night, the tests are being evaluated and changes may be made to make the tests more rigorous. These tests may be viewed at https://gitlab.e4s.io/uo-public/tau/-/pipelines. Test Smoke Durability Website Documentation Repository Spack E4S Suite Test. TAU Documentation Repository ✅ Spack ✅ ✅ ✅ 11 STEP CY2025 Report v1.1 Narrative Support for selective instrumentation for Plan Build Release Operate targeted instrumentation of DSOs in TAU. Code Test Deploy Monitor Support for CI/CD with MPI on Intel GPUs. Plan Build Release Operate Code Test Deploy Monitor Support for CI/CD with MPI on AMD GPUs. Plan Build Release Operate Code Test Deploy Monitor Support for CI/CD with MPI on NVIDIA GPUs. Plan Build Release Operate Code Test Deploy Monitor Integration of TAU in E4S and Spack. Plan Build Release Operate Code Test Deploy Monitor There were no changes in STEP personnel in CY25. STEP and the DOE Office of Science is currently funding TAU at 0.13 FTE. Kevin Huck, a key member of the TAU team left to join AMD as OASIS, RAPIDS SciDAC, and ASC/NNSA/LLNL awards to UO were terminated due to funding challenges. Although Kevin Huck wasn't working on STEP activities, we would like to note that he left the team due to funding reasons. The TAU team has worked closely with the E4S and Spack teams under the CASS ecosystem and is well aligned with the mission and needs of STEP and CASS. A joint HPCToolkit and TAU tutorial was conducted at the Energy HPC Conference in Houston in CY25 and a TAU tutorial was held at ATPESC in CY25 and we hope to deliver another tutorial at ATPESC in CY26. Further integration with DyninstAPI, PAPI, and joint tutorials with HPCToolkit are planned. In CY2026, TAU will improve on the instrumentation capabilities with DyninstAPI to support selective instrumentation with DSOs and improve on extent of CI/CD testing with GPUs from three vendors. The following deliverables and milestones are planned: 1. Demonstration of support for targeted instrumentation of DSOs with DyninstAPI at the routine level in an HPC application. 2. Support for Kokkos runtime in nightly testing of TAU with Intel GPUs with PC sampling at the CPU level and level zero instrumentation at the GPU level. 3. Support for Kokkos runtime in nightly testing of TAU with AMD GPUs with PC sampling at the GPU level. 4. Support for Kokkos runtime in nightly testing of TAU with NVIDIA GPUs with PC sampling at the GPU level. 12 STEP CY2025 Report v1.1 Narrative Note that while TAU is a mature toolkit supporting GPUs from three vendors, they are making changes to their GPU runtimes and introducing new APIs. AMD has introduced the rocprofiler-SDK in ROCm 7.2.0 and this has necessitated a re-engineering of TAU’s measurement layer significantly. Initial support for PC sampling on MI300A has been demonstrated, but it does not support simultaneous access to hardware performance counters natively in TAU yet. Future releases of ROCm may necessitate updates to support these capabilities. CI/CD tests with the Kokkos runtime will further enhance the stability and portability of TAU. Binary rewriting of shared libraries and executables to improve TAU’s usability requires additional effort. Integration with CI systems for improving TAU’s software quality is key to its usage. Porting TAU to evolving platforms and runtime systems remains an important goal to sustain the tool. 2.5 Darshan Darshan is a lightweight I/O characterization tool that captures concise summaries of I/O behavior for HPC applications. It provides detailed insights into application usage across various layers of the I/O stack. Darshan is popular among HPC application and I/O library developers, system administrators, and other I/O practitioners as it’s able to seamlessly quantify I/O characteristics of their jobs. Context To provide comprehensive understanding of HPC I/O behavior, tools like Darshan must capture details about I/O workloads across the entire storage stack (including high-level libraries, low-level system interfaces, and file system-specific interfaces) and summarize/present this information to users. The focus of this year's work has been on improving sustainability and documentation practices, and the development of a methodology for publication of Darshan logs from production applications. These developments are crucial as they facilitate understanding application I/O behavior to inform storage system research and optimization strategies. Develop processes to anonymize/publish Darshan log data captured in production for sharing with the broader HPC I/O research community, modernize and improve Darshan's documentation and release processes, and productize Darshan mechanisms for instrumenting I/O workloads at thread-level granularity. We will work with the STEP leadership to refine our metrics and identify methods to clearly demonstrate how our efforts lead to measurable increases. Property State Goal: Develop processes to anonymize/publish Darshan log data captured in production Impact Accomplishments: • Developed a methodology for continuous anonymization and publication of Darshan instrumentation data from production applications at ALCF Polaris. 13 STEP CY2025 Report v1.1 Narrative Property State Goal: Improve sustainability and documentation practices Accomplishments: • Modernized and improved documentation through migration to Read the Docs, providing up-to-date installation, deployment, and analysis guidance. Sustainability • Improved release processes to use automation to maintain consistency, with documentation automatically regenerated and associated with the appropriate release number each time a new commit is pushed to Darshan's repository. Goal: Productize Darshan mechanisms for instrumenting I/O workloads at thread-level Quality Accomplishments: • Added a thread-level information to Darshan DXT to better understand I/O usage in multithreaded applications. Test Smoke Durability Website Documentation Repository Spack E4S Suite Test Darshan Documentation Repository ✅ Spack ✅ ✅ Developed processes to Plan Build Release Operate anonymize/publish Darshan log data Code Test Deploy Monitor captured in production Modernized and improved Darshan's Plan Build Release Operate documentation and release processes Code Test Deploy Monitor Extended Darshan mechanisms for Plan Build Release Operate instrumenting I/O workloads at thread- Code Test Deploy Monitor level granularity Funding levels for Darshan in the STEP project are only sufficient for providing partial support for a single developer. However, the Darshan team provides mentoring to junior staff, hosts student interns when possible, and makes sure to promote diversity and inclusivity when possible. 14 STEP CY2025 Report v1.1 Narrative In terms of personnel, the STEP co-PI role was successfully transitioned from Phil Carns to Orcun Yildiz in November 2025, and the Darshan lead developer role was successfully transitioned from Shane Snyder to Wei-keng Liao in June 2025. Darshan has integrated well into the STEP and CASS ecosystems, providing the tooling needed for understanding and improving HPC storage access, a critical bottleneck for data-intensive applications. As a widely adopted tool for I/O characterization, Darshan offers a common framework for understanding and optimizing I/O performance across the scientific community. Collaboration between STEP software projects and CASS enables collective efforts in uncovering and addressing issues related to sustaining scientific software, particularly from the perspective of software tools. Aligned with the Genesis mission and feedback from the midterm review, this year’s milestones will focus on strengthening Darshan’s support for AI/ML workloads: Impact: Develop processes to extend Darshan’s characterization capabilities for AI/ML workloads in collaboration with application teams (e.g., ModCon, HEP-CCE) Sustainability: Conduct systematic overhead studies to quantify and minimize Darshan’s performance impact across AI/ML workloads Quality: Continue addressing software bugs to maintain Darshan’s reliability 3. Aggregated Project Status 3.1 Project Impact on HPC Community Our expansive vision is to grow the scope of STEP by fostering engagement between community stakeholders and vendors to codesign hardware monitoring interfaces and software that affect tools, with the aim of enhancing efficient utilization of low-level performance monitoring and analysis capabilities. Moreover, we seek to support work through ongoing collaborative efforts by cross-cutting teams to address shared concerns across multiple tools and enhance long-term sustainability. We also wish to support the incubation of new technology, as well as agile, short-duration activities with high potential to address rapidly evolving needs and opportunities. Finally, to the extent possible, we aim to support community-wide stewardship and advancement activities with broad impact on the tools ecosystem, e.g. standards, with the goal of enhancing performance portability and enabling efficient use of diverse hardware platforms. Given the constant rate of technical advances in high-performance computers, our topic area demands constant interactions with both vendors and users of leadership-class machines. While such technology tracking presents significant challenges, it also offers important opportunities -- especially for engagement. Given our deep understanding of the latest hardware and system-level software, we can be an asset to many who benefit from this expertise. Working together, we aim to develop a proactive stance toward the rapidly evolving hardware landscape. One example is the HPCToolkit collaboration with the Ellora Project. The Ellora project is an LLNL-led project that leverages AI for productive and trustworthy HPC software. The sophisticated measurement and analysis capabilities of HPCToolkit are being used to teach Large Language Models (LLMs) how to rewrite HPC software for improved performance and power efficiency. At the consortium level, STEP played a key role in the formulation of the Consortium for the Advancement of Scientific Software. Terry Jones is a founding member of the steering committee, and Mike Jantz is active in CASS as well. STEP also leads two CASS activities (including the vendor forum) and participates 15 STEP CY2025 Report v1.1 Narrative in several others in support of cross-cutting efforts within the consortium. In addition, STEP members and software packages are pursuing various activities to bolster the tools community, including: • Hackathons at ALCF, LBL, and OLCF • Multiple events including SIAM, ICPP, and the annual SC Conference. STEP roles included technical talks, a tools panel, a closely aligned workshop (SQCS), and BoFs. • The 2024 and 2025 Scalable Tools Workshop at Lake Tahoe • Active participation in international collaborations including ADAC and DOE-MEXT. 3.2 Analysis of Strengths, Weaknesses, Opportunities and Threats Strengths – What we do well collectively Weaknesses – Areas Needing Improvement or • Strong team of lab scientists and university Lacking Resources researchers that are leading the development • Early access to new hardware and software of impactful and well-established tools • Allocations on leadership-class machines • History of strong execution • multi-party NDA agreements with vendors • Technical prowess and highly skilled • Lacking breadth of tools workforce ○ Memory-centric tools are needed • Clear mission and goals ○ Flexible and portable capabilities for managing system power and providing quality of service are also necessary Opportunities – Favorable Trends and Threats – Challenges Faced by Perf Tools Underserved Areas to Leverage • Rapidly changing HPC systems and hardware • AI/ML (both incorporation and integration) are becoming more complex with more • Needs of cloud computing (especially with heterogeneous and distributed architectures DOE's new Lux and Solstice machines) • Rising reliance on AI/ML for modeling and • Tools to find the power used by decision-making is increasing demands for people/apps/workflows limited computational resources (especially • Ties with universities enable workforce memory and network resources) attraction • Funding levels have resulted in dramatic workforce restructuring. Current funding levels do not keep pace with rapidly shifting application domains and emerging technologies. • Software diversity in OS versions, programming languages, environments and libraries 4. Report Conclusion The Software Tools Ecosystem Project (STEP) is a collaborative initiative funded by the U.S. Department of Energy to maintain and advance critical software for high-performance computing. This annual report presents the progress of five core performance tools—HPCToolkit, PAPI, Dyninst, TAU, and Darshan—which enable the monitoring, analysis, and diagnosis of complex scientific applications. The project focuses on adapting these tools to modern GPU-accelerated architectures and emerging AI/ML workloads while fostering a skilled workforce. By coordinating between national laboratories, universities, and industry partners like NVIDIA and AMD, STEP ensures that researchers can optimize code efficiency across exascale supercomputers. The report outlines significant technical milestones, such as improved hardware counter support and automated quality assurance, alongside strategic goals for long-term software sustainability. Despite its successes, the report highlights that stable funding and early access to hardware remain vital for keeping pace with rapidly evolving technology. 16 STEP CY2025 Report v1.1 Appendix 1 – Oversight Council Appendix 1 – STEP Oversight Council Council Purpose STEP consists of an accomplished team of HPC tools professionals and representatives from vendors, facilities, and application teams. The team members include a number of representatives of widely used tools at HPC sites, including application profiling, sub-system, and system monitoring tools; foundational data interfaces; performance analysis tools, and more. They have substantial expertise and multiple years of experience in all aspects of HPC tool development, including developing for high-fidelity data concurrent and evolving types of processors, accelerators, filesystems, high speed networks; developing interactivity with a variety of software stacks and resource managers; developing interoperability among tools; and developing performance and tool output visualizations. Together, this team forms the STEP Oversight Council. The purpose of this council is to provide perspective and input, and to be a communication nexus between the STEP project and representatives of the broader HPC community. Members participate in standing meetings and are on an internal mailing-list. Council Members NAME PERSPECTIVE NAME ORGANIZATION-------- Lead PI .............. Tools .................... Terry Jones .................................... Oak Ridge National Lab Deputy PI .......... Tools ..................... Mike Jantz ..................................... University of Tennessee Co-PI ................ Tools ..................... James Brandt ........................ Sandia National Laboratories Co-PI ................ Vendors ................. James Custer .......................... Hewlett Packard Enterprise Co-PI ................ Tools ..................... Ann Gentile ........................... Sandia National Laboratories Co-PI ................ Facilities ................ Richard Gerber ..................Lawrence Berkeley National Lab Co-PI ................ Facilities ................ Kevin Harms..................................... Argonne National Lab Co-PI ................ Tools ..................... Heike Jagode ................................ University of Tennessee Co-PI ................ Tools ..................... Matthew Legendre ................. Lawrence Livermore Natl Lab Co-PI ................ Vendors ................. John Linford .......................................................... NVIDIA Co-PI ................ Vendors ................. Keith Lowery ............................... Advanced Micro Devices Co-PI ................ Facilities ................ Verónica Melesse Vergara .............. Oak Ridge National Lab Co-PI ................ Tools ..................... John Mellor-Crummey ................................. Rice University Co-PI ................ Tools ..................... Barton Miller ................................. University of Wisconsin Co-PI ................ Vendors ................. José Moreira ............................................................... IBM Co-PI ................ Applications ........... Erdal Mutlu .............................. Pacific Northwest Natl Lab Co-PI ................ Facilities ................ Suzanne Parete-Koon .................... Oak Ridge National Lab Co-PI ................ Tools ..................... Sameer Shende ..................................... University Oregon Co-PI ................ Tools ..................... Galen Shipman.................. Los Alamos National Laboratory Co-PI ................ Tools ..................... Devesh Tiwari ................................ Northeastern University Co-PI ................ Applications ........... Theresa Windus.................................... Ames National Lab Co-PI ................ Tools ..................... Orçun Yildiz ............................ Argonne National Laborator 17 STEP CY2025 Report v1.1 Appendix 2 – Glossary Appendix 2 – Glossary AI...............Artificial Intelligence ALCF .......... Argonne Leadership Computing Facility AMD ........... Advanced Micro Devices ANL ............ Argonne National Laboratory API.............. Application Programmers Interface ATPESC ...... Argonne Training Program on Extreme-Scale Computing CASS .......... Consortium for the Advancement of Scientific Software CI ............... Continuous Integration CI/CD ......... Continuous Integration/Continuous Deployment CPU ............ Central Processing Unit CUDA ......... Compute Unified Device Architecture CUPTI ......... CUDA Performance Tools Interface DAOS.......... Distributed Asynchronous Object Storage DOE............ Department of Energy DSO............ Dynamic Shared Object E4S ............. Ecosystem for Science ECP ............ Exascale Computing Program GPU ............ Graphical Processing Unity HPC............ High Performance Computing HPE ........... Hewlett Packard Enterprise HPSF .......... High Performance Software Foundation IBM ............. International Business Machines I/O .............. Input/Output LANL........... Los Alamos National Laboratory LLNL ........... Lawrence Livermore National Laboratory LLVM .......... A collection of modular and reusable compiler and toolchain technologies MEXT .......... Japanese Ministry of Education, Sports, Science and Technology ML .............. Machine Learning NDA............ Non-disclosure Agreement NERSC........ National Energy Research Scientific Computing Center NGSST ........ Next Generation Scientific Software Technologies NREL .......... National Laboratory of the Rockies OLCF .......... Oak Ridge Leadership Computing Facility OpenMP ..... Open Multi-processing. A widely-used API that simplifies writing parallel programs. ORNL.......... Oak Ridge National Laboratory PAPI............ Performance Application Programming Interface PDT ............ Program Database Toolkit PTI-GPU ...... Profiling Tools Interfaces for GPU ROCm ........ AMD's open-source stack designed for GPU computing ROI ............. Return On Investment SAST ........... Static Application and Analysis Testing SIMD .......... Single Instruction, Multiple Data SIMT ........... Single Instruction, Multiple Threads STEP ........... Software Tools Ecosystem Project TAU ............ Tuning and Analysis Utilities VM .............. Virtual Machine 18