The Impact of Documentation on Test Engagement in Pull Requests in OSS Conference: 34th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering; July 05–09, 2026; Montreal, QC, Canada 34th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (FSE Companion ’26), July 05–09, 2026, Montreal, QC, Canada DOI: 10.1145/3803437.3805555 ISBN: 979-8-4007-2636-1/2026/07 CCS: Software and its engineering Documentation CCS: Software and its engineering Software testing and debugging
Teal Amore, Nathan Berman, Siyuan Jiang · 2026
Read The Impact of Documentation on Test Engagement in Pull Requests in OSS Conference: 34th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering; July 05–09, 2026; Montreal, QC, Canada 34th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (FSE Companion ’26), July 05–09, 2026, Montreal, QC, Canada DOI: 10.1145/3803437.3805555 ISBN: 979-8-4007-2636-1/2026/07 CCS: Software and its engineering Documentation CCS: Software and its engineering Software testing and debugging by Teal Amore, Nathan Berman, Siyuan Jiang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23048.txt.
Introduction
Amore, Berman, and Jiang ask whether documentation on testing can proactively encourage coders to test code changes in open-source projects. They introduce the Test Engagement Ratio for testing frequency, using data from 160 open-source repositories. Documentation breadth shows a weak but significant positive link with testing, growing stronger in projects with more pull requests. Guides on running and writing tests correlate most strongly. The ratio also tracks test code ratio somewhat, supporting the new measure's validity.
Catalog record
| CREATOR | Teal Amore, Nathan Berman, Siyuan Jiang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21846 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.23048
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Impact of Documentation on Test Engagement in Pull Requests in OSS Conference: 34th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering; July 05–09, 2026; Montreal, QC, Canada 34th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (FSE Companion ’26), July 05–09, 2026, Montreal, QC, Canada DOI: 10.1145/3803437.3805555 ISBN: 979-8-4007-2636-1/2026/07 CCS: Software and its engineering Documentation CCS: Software and its engineering Software testing and debugging Teal Amore, Nathan Berman, Siyuan Jiang Abstract . Automated testing is crucial for maintaining open-source software quality. However, motivating contributors to include tests for code changes remains a challenge. While existing interventions, such as code coverage metrics and reviewer feedback, are often reactive and applied only after a pull request is opened, this study investigates whether documentation on testing can serve as a proactive measure to encourage testing behavior. In this work, we investigate the relationship between documentation on testing and contributor testing behavior. We introduce the Test Engagement Ratio (TER) to help understand testing frequency. Using data from 160 OSS repositories, we analyze the relationship between documentation comprehensiveness and TER. Our results show a weak but statistically significant positive correlation ( ρ = 0.36 \rho=0.36 , p < 0.001 p<0.001 ), which strengthens to a moderate relationship ( ρ = 0.44 \rho=0.44 ) in repositories with higher pull request activity. Documentation categories such as How to Run Tests and How to Write Tests show the strongest correlation with testing engagement. Furthermore, TER is found to be moderately correlated ( ρ = 0.52 \rho=0.52 , p < 0.001 p<0.001 ) with Test Code Ratio, providing preliminary evidence of its validity. Our findings suggest that documentation on testing may be associated with increased testing engagement. Future work will explore causality, documentation quality at a granular level, and cross-repository exposure effects. † † cc-license: by 1. Introduction For open-source software (OSS) repositories, pull requests that contain tests are often highly valued and more likely to get merged ( 10 ) . Despite this, motivating contributors to write tests remains challenging. Most interventions to promote testing, like code coverage metrics or reviewer comments, are reactive because they occur only after a contributor opens a pull request. These reactive approaches fail to consistently educate contributors about the project-specific testing standards. We investigate whether documentation on testing can serve as a proactive measure to encourage contributors to test more frequently. OSS repository maintainers often hold that automated testing is essential for maintaining quality, due to time limitations commonly associated with manual testing methods ( 15 ) . However developers often differ in their opinions about testing ( 6 ) ( 12 ) ( 18 ) , which impacts their engagement with it. Research suggests that testing earlier in the development process can reduce the costs associated with bug fixing ( 17 ) , motivating exploration of proactive approaches. Documentation on testing may act as proactive guidance, yet no prior work has explored whether it correlates with testing behavior. This study examines whether documentation on testing correlates with contributor testing behavior in open-source software. We build on Falcucci et al.’s dataset ( 7 ) , which categorizes contribution documentation into various testing types like How to Run Tests, Unit Testing, and Integration Testing. The number of present categories provides a measure of documentation comprehensiveness. We propose a metric test engagement ratio (TER) as a scalable proxy for testing behavior, measuring the proportion of production-code pull requests that also modify test files. We address the following research questions: • RQ1. Do contribution guidelines with more testing categories present correlate with a higher proportion of test-including pull requests? • RQ2. What documentation categories or repository metadata correlate with test-including pull requests? • RQ3. Does the proportion of test-including pull requests positively relate to standard test quality metrics? Summary of Findings Our analysis of 160 OSS repositories yields the following key results: • RQ1. We find a weak positive correlation between documentation comprehensiveness and TER ( \rho=0.36 , p<0.001 ). This strengthens to a moderate correlation ( \rho=0.44 ) in repositories with higher pull request activity. • RQ2. Documentation on How to Run Tests ( \rho=0.34 , p<0.001 ) and How to Write Tests ( \rho=0.36 , p<0.001 ) shows the strongest correlation with TER. Repository popularity metrics (stars, forks, watchers) show no significant relationship. • RQ3. TER has a moderate positive correlation with Test Code Ratio ( \rho=0.52 , p<0.001 ), providing preliminary evidence of its validity as a measure of testing activity. 2. Methodology To investigate the relationship between documentation on testing and contributor testing behaviors, we analyzed the merged pull requests from the repositories in Falcucci et al. ( 7 ) . To identify which pull requests interact with both production code and tests, we developed file classification heuristics. We then propose a testing frequency metric and conduct correlation analyses between this metric and testing categories that are present in documentation. 2.1. File Classification We classified files as test or production code using path-based heuristics from Madeja et al. ( 11 ) and extended it to include spec files, which are common in JavaScript test files. 2.2. Test Engagement Ratio (TER) Calculating traditional testing metrics presents challenges when analyzing a large dataset of repositories. While code coverage is often measured by both developers and researchers, calculating such a metric is difficult due to the challenges of successfully building and running the tests for a repository ( 3 ) . Due to these challenges, we propose a new metric, Test Engagement Ratio (TER), which enables us to measure whether a contributor has engaged in testing without having to successfully build or run the tests for each repository. We use the interaction with a test file as a proxy for engaging in testing behaviors. Unlike traditional metrics, TER does not capture whether tests are executed or meaningful, only whether contributors interact with test artifacts. We define the Test Engagement Ratio as: \text{Test Engagement Ratio = }\frac{\sum_{r\in R}I_{\text{both}}(r)}{\sum_{r\in R}I_{\text{prod}}(r)} where R is the set of pull requests for a repository and the indicator functions are defined as: I_{\text{prod}}(r)=\begin{cases}1&\text{if PR $r$ modifies a production file},\\ 0&\text{otherwise}.\end{cases} I_{\text{both}}(r)=\begin{cases}1&\text{if PR $r$ modifies both production and test files},\\ 0&\text{otherwise}.\end{cases} In words, TER is the ratio of pull requests that modify both production and test files to those that modify production files, capturing the likelihood that contributors who change production code also modify tests. 2.3. Test Documentation Diversity Score To quantify the comprehensiveness of a repository’s documentation on testing, we define the test documentation diversity score. This score is calculated as the number of testing categories present in the repository’s contribution documentation. Each category that is present contributes one point to the score. The 10 possible categories are detailed in Table 2. The range of possible scores is from 0 to 10, where a higher score indicates more comprehensive documentation on testing. 2.4. Data Collection Using the file classification heuristics defined above, we developed a tool to gather all merged pull requests from January 1, 2025 through December 1, 2025. For each pull request, we recorded whether it modified a test file or a production code file. From this data, we calculated each repository’s TER for use