Gamifying Testing in IntelliJ: A Replicability Study DOI: 10.1145/3728983 Journal: PACMSE Volume: 2 ISSTA ISSTA106 7 CCS: Software and its engineering Software testing and debugging CCS: Software and its engineering Integrated and visual development environments
Philipp Straubinger, Tommaso Fulcini, Giacomo Garaccione, Luca Ardito, Gordon Fraser · 2025
Read Gamifying Testing in IntelliJ: A Replicability Study DOI: 10.1145/3728983 Journal: PACMSE Volume: 2 ISSTA ISSTA106 7 CCS: Software and its engineering Software testing and debugging CCS: Software and its engineering Integrated and visual development environments by Philipp Straubinger, Tommaso Fulcini, Giacomo Garaccione, Luca Ardito, Gordon Fraser (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-19294.txt.
Introduction
Philipp Straubinger and four co-authors ran a repeat study in twenty twenty-five on making software tests more fun. The tool is IntelliGame, a plugin for the IntelliJ IDEA coding app. It gives points and feedback to push coders to write and run more tests.
Past work hinted that such game-like rewards help, but proof was thin. This study used one hundred seventy-four people in two groups, one with the plugin and one without. It also moved to a new language, TypeScript, and a larger pool. The plugin group wrote more tests, ran them more often, and used test tools more. Their code came out better, and the team reads this as proof the effect is real.
Catalog record
| CREATOR | Philipp Straubinger, Tommaso Fulcini, Giacomo Garaccione, Luca Ardito, Gordon Fraser |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-19294
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Gamifying Testing in IntelliJ: A Replicability Study DOI: 10.1145/3728983 Journal: PACMSE Volume: 2 ISSTA ISSTA106 7 CCS: Software and its engineering Software testing and debugging CCS: Software and its engineering Integrated and visual development environments Philipp Straubinger, Tommaso Fulcini, Giacomo Garaccione, Luca Ardito, Gordon Fraser Abstract . Gamification is an emerging technique to enhance motivation and performance in traditionally unengaging tasks like software testing. Previous studies have indicated that gamified systems have the potential to improve software testing processes by providing testers with achievements and feedback. However, further evidence of these benefits across different environments, programming languages, and participant groups is required. This paper aims to replicate and validate the effects of IntelliGame , a gamification plugin for IntelliJ IDEA to engage developers in writing and executing tests. The objective is to generalize the benefits observed in earlier studies to new contexts, i.e., the TypeScript programming language and a larger participant pool. The replicability study consists of a controlled experiment with 174 participants, divided into two groups: one using IntelliGame and one with no gamification plugin. The study employed a two-group experimental design to compare testing behavior, coverage, mutation scores, and participant feedback between the groups. Data was collected through test metrics and participant surveys, and statistical analysis was performed to determine the statistical significance. Participants using IntelliGame showed higher engagement and productivity in testing practices than the control group, evidenced by the creation of more tests, increased frequency of executions, and enhanced utilization of testing tools. This ultimately led to better code implementations, highlighting the effectiveness of gamification in improving functional outcomes and motivating users in their testing endeavors. The replication study confirms that gamification, through IntelliGame , positively impacts software testing behavior and developer engagement in coding tasks. These findings suggest that integrating game elements into the testing environment can be an effective strategy to improve software testing practices. 1. Introduction In software development, ensuring the quality and reliability of software products is a key priority. Software testing plays a crucial role in this process, serving as a core element of the software development lifecycle ( Tuteja et al. 2012 ) . It helps identify defects, improve functionality, and ensure that applications meet their intended requirements. Despite its importance, however, software testing is often seen as one of the less appealing phases of development, which can lead to testers feeling undervalued or disengaged ( Straubinger and Fraser 2023 ) . This lack of motivation can hinder productivity and negatively impact the overall quality of the software produced ( Paiva et al. 2010 ) . To address these challenges, recent literature has explored the concept of gamification , a technique that applies game-design elements in non-game contexts ( Deterding et al. 2011 ) . Across various aspects of Software Engineering gamification has shown promising results in terms of engagement and performance ( Fulcini et al. 2023 ) . There have been several attempts to incorporate gamification into the software testing process, both to enhance learning (especially in academic settings) and to improve the practical execution of testing. Gamified tools and environments have been applied to a range of testing activities, including unit testing and GUI testing ( Coppola et al. 2024 ; Materazzo et al. 2023 ; Fulcini and Ardito 2022 ) , test creation, execution, and maintenance ( Rojas and Fraser 2016 ; Straubinger and Fraser 2024c ; Straubinger and Fraser 2024b ) . Most studies in this area have focused on exploring new tools and conducting preliminary evaluations to validate their effectiveness and user experience (UX) in small-scale contexts ( Fulcini et al. 2023 ) . So far, these gamified approaches have largely been developed independently, with limited efforts to generalize the results across different contexts. However, there is a need for consolidation of the findings, requiring an effort to replicate the existing validation studies in other contexts, to achieve higher confidence in the obtained results and move towards well-established understanding. With this goal in mind, we conducted a replication study to assess one of the latest gamified tools for unit testing: IntelliGame , a gamified plugin for the IntelliJ IDEA Integrated Development Environment (IDE) that rewards testers with achievements for good testing practices ( Straubinger and Fraser 2024c ) . Given the promising results of the initial evaluation, this tool merits further exploration, as limited participation may restrict the generalizability of observed benefits. This paper presents our attempt to adapt IntelliGame for broader use, validating it through a controlled empirical experiment aimed at generalizing its effectiveness. The key contributions of this paper are as follows: • We present an empirical experiment to validate the existing IntelliGame tool using a two-group experimental design on a new programming language with a sample of 174 participants. • We describe a statistical analysis of participants’ behavior and performance in writing and executing tests. • We present a comparison of the findings from the original study with those of our study. Our replication confirms that gamification can effectively impact user behavior by encouraging the creation of more tests, more frequent test executions, and increased use of testing tools, like coverage reports and debugging tools. We observed that while both groups exhibited similar testing approaches, gamified participants wrote more tests, focused on finer details, and produced code with fewer failing tests of the reference test suite—implying a slightly better functional outcome. Additionally, we found that achievement levels in IntelliGame correlated positively with both tester motivation and the quality of test suites. These results align with several conclusions from the original study, specifically that gamification can increase testing frequency and improve user engagement with tools for quality assurance. However, our study adds nuances. For instance, we found that IntelliGame also impacts debugging behavior—a result not highlighted in the original study. Further introspection on test quality and individual achievements indicates that while gamification users created more detailed test cases, their increased focus on quantity occasionally led to higher failure rates and test smells, particularly in the area of date and time handling. This indicates that while gamification encourages thoroughness, the IntelliGame approach would benefit from additional or modified achievements aimed at quality to balance the focus on test quantity. Further, unlike the original study, our participants in the treatment group reported greater time pressure and showed a greater awareness of the quality of their tests compared to the control group, likely due to the continuous feedback provided by IntelliGame . In summary, we successfully replicated the original study on a gamification plugin with achievement features in the integrated development environment, largely confirming its results. 2. Background 2.1. Gamification in Software Testing Gamification is a technique adopted for increasing motivation and interest in unappealing tasks and is considered as the usage of game elements in non-recreational contexts ( Deterding et al. 2011 ) . An additional benefit is that tasks performed in a gamified environment can produce better output compared to non-gamified equivalent tasks ( de Paula Porto et al. 2021 ; Stol et al. 2021 ) . Among the various game elements, the most commonly adopted ones in gamified approaches are points, leaderboards, badges, and awards ( Barreto and França 2021 ; de Jesus et al. 2018 ) , with other possible elements such