Exploring the Output of Software Testing Tools through a Visual Comparative Analysis DOI: XXXXXXX.XXXXXXX Conference: ; ; CCS: Human-centered computing Interaction tech
Brandon Lit, Anthony Maocheia-Ricci, Thomas Driscoll · 2026
Read Exploring the Output of Software Testing Tools through a Visual Comparative Analysis DOI: XXXXXXX.XXXXXXX Conference: ; ; CCS: Human-centered computing Interaction tech by Brandon Lit, Anthony Maocheia-Ricci, Thomas Driscoll (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-04189.txt.
Introduction
Brandon Lit, Anthony Maocheia-Ricci, and Thomas Driscoll compare how software testing tools show their results to the people who run them. Their study covers fifty tools and test harnesses across four well used programming languages, most with text output and a few with visual screens.
The team records the shared parts of these outputs, how results are laid out, and how colour is used in both text and visual tools. Their findings point to clear trends that tool makers can follow, so test results become easier to read, scan, and act on.
Catalog record
| CREATOR | Brandon Lit, Anthony Maocheia-Ricci, Thomas Driscoll |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.04189
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Exploring the Output of Software Testing Tools through a Visual Comparative Analysis DOI: XXXXXXX.XXXXXXX Conference: ; ; CCS: Human-centered computing Interaction tech Brandon Lit, Anthony Maocheia-Ricci, Thomas Driscoll Abstract . Software testing is a fundamental process of software development, and prior work has shown that visualizations of test results support testers’ decision-making. However, Human-Computer Interaction research on software testing has yet to explore and understand the shared interface elements and patterns in visualization of testing outputs. To address this, we conducted a visual comparative analysis of the output of 50 software testing tools and harnesses (44 with CLI output, 6 with GUI output) across four popular programming languages. Our analysis reveals the common interface elements in software testing tools, how these tools display and visualize test results, as well as the specific make-up of the output. Our findings provide insight on how visual testing output is formatted and how colour is used across both CLI and GUI environments, identifying trends that can be applied by developers of testing tools. 1. Introduction Software testing is a fundamental process of the software development pipeline. Much empirical research in software testing explores areas such as evaluation metrics ( 6 ) or automated testing methods or frameworks (e.g., fuzz testing ( 17 ; 7 ) , the use of Large Language Models ( 33 ; 32 ; 24 ) ), but little work has explored the outputs of these testing tools and harnesses with respect to the end-user’s interactions. While some software testing tools’ output exists only as file structure (e.g., YARPGen ( 20 ) ), other platforms include visual output through dashboard websites, or are dedicated GUI tools themselves. Studies in software testing show how processes benefit from having visualizations of test results ranging from a colour gradient to denote bug severity to full graphical user interface (GUI) dashboards ( 30 ) . Other qualitative studies in software engineering related to software testing point towards the usefulness of visualizations ( 12 ) or graphical outputs of the tests, their coverage, and properties of interest ( 25 ) , which can serve as decision support tools for software engineers ( 29 ) . Human-Computer Interaction (HCI) research is well-equipped to explore this space of end-user interactions with software testing tools. Existing HCI studies related to software testing focus primarily on accessibility ( 8 ) and GUI testing tools or platforms ( 1 ; 5 ; 19 ; 23 ) . However, building an understanding of these tools in terms of their shared interface elements and patterns in visualization has not yet been explored. As such, to further bridge the gap between HCI and software testing, we employ a visual comparative analysis methodology to explore the visual output of software testing tools and harnesses as a whole. Visual Methods are a family of qualitative methodologies primarily used in the social sciences to critically interpret visual materials such as fine artwork, maps, and digital images or photography ( 26 ) . We adapt prior work using visual methods in HCI research ( 11 ) as a basis for rigor in our methodology. With this, we aim to explore the following research questions: RQ1: What are the shared design patterns and classes of visual output from software testing tools? RQ2: How are various testing-related metrics displayed or visualized in this visual output? Through this work, we visually examined the output of 50 software testing tools and harnesses (44 with CLI output, 6 with GUI output) across four popular programming languages, discovered shared visual elements found in outputs, cross-compared their visual layout, and explored how common interface elements (status details and test suite summaries) display and visualize important information. Drawing from these findings, we discuss the use of colour and GUIs in testing outputs to convey information and identify visual trends for developers of testing harnesses. 2. Related Work To our knowledge, no other work exists in the visual analysis of software testing tool output. As such, we explore existing work at the intersection of HCI and software testing and the use of visual methodologies within HCI to ground our work. 2.1. HCI and Software Testing Figure 1. Our visual comparative analysis methodology phases, adapted from Frappier et al. ( 11 ) . The visual comparative analysis methodology shown visually. The stages are presented sequentially from left to right: (1) Identification, (2) Running Programs, (3) Wireframing, (4) Visual Coding, (5) Abstracting, (6) Collating, and (7) Analyzing. Most prior work related to software testing and HCI has focused on GUI testing tools, and various platforms have been created to support playback or automation. Alaboudi and LaToza’s Hypothesizer ( 1 ) enables hypothesis-driven playback testing to aid a developer in determining the cause of a bug in a web-based system. Chen et al. ( 5 ) augment crowdsourcing-driven GUI testing by showing test traces visually in real time. For Android devices, research has explored both the augmentation of manual GUI testing through hints ( 19 ) and automatic testing through the use of UI patterns as heuristics ( 23 ) . Where these studies are primarily in creating novel tools to aid users in testing GUIs, they show the importance of visual cues and outputs in the software testing lifecycle. Additional work in software testing has adapted methods and techniques from HCI to better understand the perspective of software testers as a user group. For instance, 10 asked the question “Who is Testing?” and investigated the backgrounds of over 70 industry testers to learn about differences in needs and problem-solving preferences. 4 propose using methods from HCI to learn about the various testing techniques used and what types of bugs are found by testers with and without formal training. These works show a need for understanding of what testing tools output from a visual perspective such that future testing tools and output can be designed with these considerations in mind. 2.2. Visual Methodologies in HCI Though visual output is often analyzed in qualitative HCI and visualization work, methods to work with visual data are underdeveloped, requiring the adaptation of existing qualitative methods for textual data ( 31 ) . In the social sciences, visual research is more established, with photos and videos being integrated in ethnographic research through analysis of existing photos, photo-elicitation studies, or having participants take photos as part of research ( 27 ) . Where prior qualitative work in HCI has used photo-elicitation as a primary methodology ( 9 ; 13 ) , visual methods are not often used for analyzing existing technological artifacts. This form of analysis may aid in understanding and exploring design patterns for future work in the design of technological systems ( 31 ) . Early work in this space includes understanding common design patterns in: data comics ( 3 ) , data stories ( 34 ) , and composite visualizations ( 15 ) . An exemplar paper in the use of visual methodologies within HCI research is Frappier et al. ( 11 ) ’s visual comparative analysis of online debate platforms. In their work, they perform a 7-phase visual analysis process to compare features, understand interface patterns, and discuss design implications for future work and development in debate platforms. Our work builds