Harden and Catch for Just-in-Time Assured LLM-Based Software Testing: Open Research Challenges DOI: 10.1145/3663529.3663839 fsecomp24industry-p37-p ISBN: 979-8-4007-0658-5/24/07 Conference: Companion Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering; 23

Mark Harman, Peter O’Hearn, Shubho Sengupta · 2025

Read Harden and Catch for Just-in-Time Assured LLM-Based Software Testing: Open Research Challenges DOI: 10.1145/3663529.3663839 fsecomp24industry-p37-p ISBN: 979-8-4007-0658-5/24/07 Conference: Companion Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering; 23 by Mark Harman, Peter O’Hearn, Shubho Sengupta (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-16472.txt.

Introduction

Mark Harman, Peter O'Hearn, and Shubho Sengupta define two kinds of software tests and a challenge built on them, in a keynote paper on AI-written tests. A hardening test guards against future breakage; a catching test actually trips on a fault, whether in new code or hiding in old. Language models can now draft such tests at scale.

Their Just-in-Time challenge asks for tests written at the moment a code change lands, catching its faults before they reach production. They show a solver for that challenge could also be turned loose on legacy code to catch faults already lurking there. The paper maps the possible outcomes for each test type, lists open research problems and deployment routes, and shares early results from hardening work inside Meta. The authors frame assured, timely testing as a high-impact frontier for language models.

Catalog record

CREATORMark Harman, Peter O’Hearn, Shubho Sengupta
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE92 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-16472

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Harden and Catch for Just-in-Time Assured LLM-Based Software Testing: Open Research Challenges DOI: 10.1145/3663529.3663839 fsecomp24industry-p37-p ISBN: 979-8-4007-0658-5/24/07 Conference: Companion Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering; 23 – 27, 2025; Trondheim, Norway Companion Proceedings of the 33rd ACM Symposium on the Foundations of Software Engineering (FSE ’25), June 23–27, 2025, Trondheim, Norway CCS: Software and its engineering Software testing and debugging Mark Harman, Peter O’Hearn, Shubho Sengupta Abstract . Despite decades of research and practice in automated software testing, several fundamental concepts remain ill-defined and under-explored, yet offer enormous potential real-world impact. We show that these concepts raise exciting new challenges in the context of Large Language Models for software test generation. More specifically, we formally define and investigate the properties of hardening and catching tests. A hardening test is one that seeks to protect against future regressions, while a catching test is one that catches such a regression or a fault in new functionality introduced by a code change. Hardening tests can be generated at any time and may become catching tests when a future regression is caught. We also define and motivate the Catching ‘Just-in-Time’ (JiTTest) Challenge, in which tests are generated ‘just-in-time’ to catch new faults before they land into production. We show that any solution to Catching JiTTest generation can also be repurposed to catch latent faults in legacy code. We enumerate possible outcomes for hardening and catching tests and JiTTests, and discuss open research problems, deployment options, and initial results from our work on automated LLM-based hardening at Meta. This paper 1 1 1 Author order is alphabetical. The corresponding author is Mark Harman. was written to accompany the keynote by the authors at the ACM International Conference on the Foundations of Software Engineering (FSE) 2025. 1. Introduction For the past decade, at Meta, we (the authors) have been developing automated static and dynamic analysis and testing technologies. We have also been involved in the development of AI technologies, as a support to such analyses, and also as an independent technology in its own right. As well as our recent industrial experience, we have been active in the Software Engineering, Programming Languages, and Artificial Intelligence research communities over several decades. Our observation is that, despite much research and development, some of the most fundamental concepts that lie at the intersection between scientific research and practical software testing have yet to be adequately analysed, formalised, or defined, let alone fully tackled. Research and practice are unequivocally and rapidly moving towards automated software test design, increasingly relying on language models to generate the test cases ( mhetal:LLM-survey ) . This makes it more important than ever to have crystal clear definitions of the most pressing problems facing practising software engineers, when it comes to automated test generation ( anandetal:orchestrated ) . Automated test generation is not only important for testing code generated, both by humans and models, it may also prove useful for training and assessing the models themselves ( jain2024testgeneval ) . In this paper, we attempt to provide simple, intuitive, and precise formulations of the underlying concepts on which automated software test design rests. We first formalise the way in which generated tests harden against future regressions. This regression test generation has been the primary focus of both academic research and practice in automated test generation for the past few decades, with notable successes ( mh:icst15-keynote ; cadar:three-decades ; mhetal:ssbse18-keynote ; zalewski:afl ) . These regression tests are ‘hardening’ tests according to our terminology, because they seek to detect bugs in future changes rather than in the current system; they ‘harden’ against future regressions. Research and practice has tended to focus on regression testing due to the Oracle Problem ( ebetal:oracle ) . Regression testing offers the automated test tool designer a readily deployable solution to the Oracle Problem: the Regression Oracle ( alshahwan:software ) . The Regression Oracle dictates that the behaviour of the previous version of the system should be preservered by any change, thereby ensuring that there are no regressions. In Section 2 we briefly review our previous work on automated test generation at Meta, leading up to our current work on LLM-based automated unit test generation for hardening. The primary contributions of this work lie in the way we use LLMs and mutation. Nevertheless, by focussing solely on regression testing, we (and the wider the community), have overlooked an important untackled problem: the Catching Just-in-Time test (Catching JiTTest) problem. A JiTTest is generated when a pull request is submitted. We also formalise and analyse JiTTests. By seeking to generate a catching JiTTest, we aim to catch bugs specifically in that pull request. A JiTTest is not concerned with some possibly distant future in which some regression may (or may not) occur, but the immediate future in which a pull request is about to land into production. Generating Catching JiTTests represents a more challenging problem than generating hardening regression tests, although it subsumes that problem. It is also much more impactful. A JiTTest can catch bugs in new functionality as well as regressions, and it stops them literally just-in-time . That is, just before the pull request lands into production. Because they do not merely target regressions, JiTTests can also easily be repurposed to find latent bugs in the existing code base, as we show in Section 6.5 . We believe the Catching JiTTest Challenge to be the most challenging and impactful problem in software testing. Although there has been initial discussion of just-in-time testing approaches ( fernandez1996using ; veanes2005fly ; mossige2015testing ; brandt2022developer ) , much more remains to be done. Section 3 , introduces our formalisation of hardening and catching. Our formalism allows us to precisely determine when a hardening test subsequently catches a bug. It also allows us to distinguish between tests that catch regressions and those that catch bugs in new functionality. We use our formalism to precisely define the Catching JiTTest Challenge. Our definitions apply to testing in general, whether tests are generated by machine or by human, and irrespective of the generation technique used. Nevertheless, the motivation for our work is our recent development and deployment ( mhetal:TestGen-LLM ; foster:mutation ) of LLM-based test generation (using an Assured LLMSE approach ( mhetal:intense24-keynote ) ). This recent LLM-based deployment provides the context in which we illustrate the applications and implications of the ‘harden-and-catch’ formalism. The paper also includes a case-based analysis of possible deployments for hardening and catching tests, considering the balance of risk and reward inherent in their deployment. This analysis reveals new insights into deployment options. The primary contributions of this paper are as follows: (1) Introduction of the Catching JiTTest Challenge : Although challenging, we argue that the Catching JiTTest Challenge is the most highly impactful currently open challenge in automated test generation. We also show how advances in LLMs open the door to previously unavailable solutions, making the time ripe to tackle Catching JiTTest Challenge. (2) Precise Formal Foundations : A formalisation of hardening and catching tests, and the distinction between regression catching and functionality catching tests. (3) Detailed Practical Deployment Analysis : A case-by-case analysis of deployment options, illustrating how the formalism informs practical deployment options.

Read the complete text