Providing Information About Implemented Algorithms Improves Program Comprehension: A Controlled Experiment 149 Conference: The 29th International Conference on Evaluation and Assessment in Software Engineering; 17–20 June, 2025; Istanbul, Türkiye CCS: Software and its engineering Software maintenanc
Denis Neumüller, Alexander Raschke, Matthias Tichy · 2025
Read Providing Information About Implemented Algorithms Improves Program Comprehension: A Controlled Experiment 149 Conference: The 29th International Conference on Evaluation and Assessment in Software Engineering; 17–20 June, 2025; Istanbul, Türkiye CCS: Software and its engineering Software maintenanc by Denis Neumüller, Alexander Raschke, Matthias Tichy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-19225.txt.
Introduction
Denis Neumüller and co-authors test if labels that name the algorithm in a piece of code help readers grasp it. Their paper from 2025 is a controlled trial. Tools can spot algorithms in code on their own, but no one had checked if the labels in fact help.
Fifty six people took part. Half read code with algorithm labels that gave the name and extra facts, and half read plain code. The labeled group scored higher on grasp tasks, by a median gain the authors put at about a quarter. Time to finish did not change. Most readers called the labels helpful, above all for seeing what the code is for. Uses named ranged from spotting errors to swapping in library code.
Catalog record
| CREATOR | Denis Neumüller, Alexander Raschke, Matthias Tichy |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-19225
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Providing Information About Implemented Algorithms Improves Program Comprehension: A Controlled Experiment 149 Conference: The 29th International Conference on Evaluation and Assessment in Software Engineering; 17–20 June, 2025; Istanbul, Türkiye CCS: Software and its engineering Software maintenance tools Denis Neumüller, Alexander Raschke, Matthias Tichy Abstract . Context: Various approaches aim to support program comprehension by automatically detecting algorithms in source code. However, no empirical evaluations of their helpfulness have been performed. Objective: To empirically evaluate how algorithm labels — which include the algorithm’s name and additional information — impact program comprehension in terms of correctness and time. Method: We conducted a controlled experiment with 56 participants, where the experimental group received code with labeled algorithms. The groups completed exercises designed to measure program comprehension as well as a post-questionnaire on label helpfulness, use cases for algorithm recognition, and reasons for self-implementation of algorithms in practice. Results: Annotating source code with algorithm labels significantly improves program comprehension ( p =0.040), with a median improvement of 6 points (~23%), but does not affect completion times ( p =0.991). Qualitative analysis revealed that a majority of participants perceived the labels as helpful, especially for recognizing the codes intent. Participants also proposed use cases such as error detection, optimization, and library replacement. Reasons for self-implementing algorithms included library inadequacies, performance needs and avoiding dependencies or licensing costs. Conclusion: This study shows that algorithm labels improve program comprehension, especially for developers with medium programming experience. Our qualitative analysis also sheds light on how participants benefit from the labels, further use cases for algorithm recognition and motivations behind self-implementing algorithms. 1. Introduction Program comprehension plays a crucial role in software development, with empirical studies showing that developers spend over half of their time understanding code ( Xia et al. 2018 ; Minelli et al. 2015 ) . A variety of approaches, usually referred to as Cliché-, Plan-, Concept- or Algorithm-recogni-tion, aim to support program comprehension by automatically recognizing language-independent ideas of computation and problem-solving methods, such as algorithms and coding strategies used in source code ( Kozaczynski et al. 1992 ; Quilici 1994 ; Wills 1994 ; Metzger and Wen 2000 ; Alias and Barthou 2003 ; Zhu and Zhu 2011 ; Taherkhani and Malmi 2013 ; Mesnard et al. 2016 ; Nunez et al. 2017 ; Marin et al. 2017 ; Long et al. 2022 ; Neumüller et al. 2024 ) . However, to the best of our knowledge, none of these approaches actually perform an evaluation of their impact on program comprehension. The few evaluations that exist only consider technical aspects, such as scalability. This work’s goal is to evaluate the effectiveness of these approaches in supporting program comprehension. Given that algorithms represent some of the more intricate patterns detected by these tools, our study specifically focuses on them. We follow Cormen et al. ( Cormen et al. 2009 , p. 5) and define algorithm as “a specific computational procedure […] for solving a well-specified computational problem” , e.g., sorting, handling data structures, computing shortest paths in a graph, etc. While our study specifically focuses on algorithms, we believe the findings also provide valuable evidence for the effectiveness of recognizing computational patterns and concepts in source code, since algorithms can be seen as special types of these. To ensure a tool-agnostic evaluation, we present algorithm information in a generic way by inserting comments into the code base. These comments, referred to as algorithm labels in the following, consist of the algorithm’s name as well as a Wikipedia link for further reference. An example of our algorithm labels can be seen in . The goal of our study can be summarized using the GQM template ( Jedlitschka and Pfahl 2005 ) as follows: Analyze algorithm labels for the purpose of evaluation with respect to their influence on program comprehension from the point of view of researchers and programmers in the context of students, researchers and software developers using Java. From this goal we derive the following research questions: RQ1: Does annotating the source code with labels of the implemented algorithms improve program comprehension? RQ2: Does annotating the source code with labels of the implemented algorithms improve the time required to comprehend a program? RQ3: What qualitative insights can we gain from the participants regarding the influence of algorithm labels. To answer our RQs we conducted a controlled experiment with 56 participants, including students, researchers and industry developers. Participants were assigned to either the control group or the experimental group, which received algorithm labels as comprehension support. The groups completed exercises in a realistic Java code base designed to measure program comprehension allowing us to answer RQ1 and RQ2 . For RQ3 , we predefined 3 analysis dimensions exploring them through the post-questionnaires and interviews: ( i ) the perceived usefulness of labels ( ii ) use cases for algorithm recognition ( iii ) reasons for self-implementation of algorithms . The contributions of our paper include an evaluation of the impact of algorithm labels on program comprehension in terms of correctness and time, as well as qualitative insights into why these labels are helpful, use cases for algorithm recognition, and reasons for self-implementation of algorithms in practice. 2. Experimental Design We present the design and findings of our study following the guidelines on reporting empirical studies in Software Engineering outlined by Jedlitschka et al. ( Jedlitschka and Pfahl 2005 ) . The design and setup of our study follows the recommendations by Siegmund ( Siegmund 2012 ) and Dunsmore et al. ( Dunsmore and Roper 2000 ) , who outline best practices for designing program comprehension experiments, such as controlling confounding variables. A replication package for our work is available on Zenodo ( Neumüller et al. 2025 ) . 2.1. Goals Figure 1. Overview of the study procedure. Figure 1 gives an overview of the design of our study. As described in Section 1 the goal of our experiment was to find out if annotating source code with labels regarding the implemented algorithms improves program comprehension or the time required to comprehend programs. To investigate this we designed a controlled experiment dividing our participants into a control and experimental group. Initially, participants watched an introductory video that outlined the study procedure, including the types of exercises and the think-aloud methodology. It also introduced the CoCoME codebase (see Sec. 2.5 ), providing an overview of its structure and demonstrating how to launch the software and run its unit tests in Eclipse. We then collected demographic and educational characteristics about the participants based on a questionnaire designed by Siegmund ( Siegmund 2012 , p. 83) which we adopted almost one-to-one. Since our study also included participants from industry we added questions on job title and working experience. To assess participants’ prior programming experience, we utilized a set of programming questions modeled after those developed by Siegmund ( Siegmund 2012 , p. 193-199) . Thereafter, the actual experiment began in which the participants had to complete four exercises consisting of program comprehension and maintenance exercises. During these exercises the source code of the experimental group was annotated with the algorithm labels such as the one displayed in . After the experiment the participants had to answer a post-question-naire designed to