Just-in-Time Catching Test Generation at Meta DOI: 123456789 not-yet-known ISBN: 979-8-4007-0658-5/24/07 Conference: Companion Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering; June; Montreal Companion Proceedings of the 34th ACM International Conferenc
Matthew Becker, Yifei Chen, Nicholas Cochran, Pouyan Ghasemi, Abhishek Gulati, Mark Harman * , Zachary Haluza, Mehrdad Honarkhah, Herve Robert, Jiacheng Liu, Weini Liu, Sreeja Thummala, Xiaoning Yang, Rui Xin, Sophie Zeng · 2026
Read Just-in-Time Catching Test Generation at Meta DOI: 123456789 not-yet-known ISBN: 979-8-4007-0658-5/24/07 Conference: Companion Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering; June; Montreal Companion Proceedings of the 34th ACM International Conferenc by Matthew Becker, Yifei Chen, Nicholas Cochran, Pouyan Ghasemi, Abhishek Gulati, Mark Harman * , Zachary Haluza, Mehrdad Honarkhah, Herve Robert, Jiacheng Liu, Weini Liu, Sreeja Thummala, Xiaoning Yang, Rui Xin, Sophie Zeng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22832.txt.
Introduction
Matthew Becker and a large team at Meta describe tests that are built to fail, on purpose. Their system writes tests just in time, as new code is about to land in huge back-end systems. Normal tests pass when written; these catching tests are meant to fail and expose a bug first.
Catching a bug before code ships is far cheaper than fixing it in production. The systems involved hold hundreds of millions of lines of code, so hand testing cannot keep up. Automated catching tests give reviewers an early warning where it counts.
Catalog record
| CREATOR | Matthew Becker, Yifei Chen, Nicholas Cochran, Pouyan Ghasemi, Abhishek Gulati, Mark Harman * , Zachary Haluza, Mehrdad Honarkhah, Herve Robert, Jiacheng Liu, Weini Liu, Sreeja Thummala, Xiaoning Yang, Rui Xin, Sophie Zeng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 87 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-22832
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Just-in-Time Catching Test Generation at Meta DOI: 123456789 not-yet-known ISBN: 979-8-4007-0658-5/24/07 Conference: Companion Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering; June; Montreal Companion Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering (FSE Companion ’26), June, 2026, Montreal Thanks: * The corresponding author is Mark Harman. Author order is alphabetical. CCS: Software and its engineering Software testing and debugging Matthew Becker, Yifei Chen, Nicholas Cochran, Pouyan Ghasemi, Abhishek Gulati, Mark Harman * , Zachary Haluza, Mehrdad Honarkhah, Herve Robert, Jiacheng Liu, Weini Liu, Sreeja Thummala, Xiaoning Yang, Rui Xin, Sophie Zeng Abstract . We report on Just-in-Time catching test generation at Meta, designed to prevent bugs in large scale backend systems of hundreds of millions of line of code. Unlike traditional hardening tests, which pass at generation time, catching tests are meant to fail, surfacing bugs before code lands. The primary challenge is to reduce development drag from false positive test failures. Analyzing 22,126 generated tests, we show code-change-aware methods improve candidate catch generation 4x over hardening tests and 20x over coincidentally failing tests. To address false positives, we use rule-based and LLM-based assessors. These assessors reduce human review load by 70%. Inferential statistical analysis showed that human-accepted code changes are assessed to have significantly more false positives, while human-rejected changes have significantly more true positives. We reported 41 candidate catches to engineers; 8 were confirmed to be true positives, 4 of which would have led to serious failures had they remained uncaught. Overall, our results show that Just-in-Time catching is scalable, industrially applicable, and that it prevents serious failures from reaching production. 1. Introduction Most work on unit test generation ( 6 ; 9 ; 26 ; 50 ; 34 ; 59 ; 18 ; 62 ; 45 ) is concerned with ‘hardening’ tests. That is, tests that pass at generation time, and land into the code base to ‘harden’ against future regressions. By contrast, ‘catching’ tests fail at test generation time by design, with the aim of catching bugs in proposed changes ( 36 ) . A catching test cannot land together with the changes that it tests, since it fails on those changes. Therefore, the catching test must either be discarded (because it is deemed unhelpful), or its signal must be addressed (to fix the changes and make the test pass). We target regression catching tests that fail on the proposed changes (a code revision) but pass on the parent revision. For catching tests we cannot rely on the implicit oracle ( 12 ) . Our auto-generated tests must detect regressions according to the general oracle —the true, expected behavior, which is often ill-defined and only partly knowable ( 12 ; 69 ) . The implicit oracle merely captures errors, such as crashes, that are known to be true positive failures regardless of the specification ( 12 ; 32 ) . By contrast, the general oracle is the (usually vague and informal, even unstated) specification of correct behavior. This inherent vagueness makes it hard (yet no less important) to distinguish true from false positive test failures. A weak catch is simply one that fails on the current revision, whereas a strong catch is one that also should fail (a true positive failure) according to this general oracle. Its failure reveals a bug in the code under test. A strictly weak catch is a false positive failure that reveals a bug in the test case, including oracle misalignment. The challenge is to determine whether a weak catch is strong. The problem of automatically generating such strong catching tests Just-in-Time (JIT) as revisions are submitted, is called the ‘Catching JiTTest Challenge’, introduced in the FSE 2025 keynote paper ( 36 ) . At Meta, a revision submitted to the Continuous Integration (CI) system is called a ‘diff’ (short for differential). Our catch generation workflows are deployed at diff submission time. We focus on the potential for a diff to introduce a severe regression. It is important to focus on severe regressions, because catching test generation is inherently computationally expensive; many tests must be generated to increase the chances of finding a strong catch. We prioritize catching severe bugs using a targeting mechanism similar to Diff Risk Score (DRS) ( 1 ) , trained on previous code changes and running overnight (west coast USA) using spare capacity. The workflow is triggered for all the diffs submitted for the previous day that are flagged as high risk by the targetter. We seek to answer the following question for automated software test generation: How can we automatically generate tests that find the most severe bugs in proposed changes, before the bugs cause severe failures and with high enough true positive rate that the test signal does not impede engineer velocity? We deployed two catch generation workflows and three true/false positive assessors. We used them to test Meta’s systems of hundreds of millions of lines of code used by over 3.5 billion people worldwide every day. Our primary findings are as follows:- Effectiveness of diff-awareness : Based on a study of 22,126 tests generated, we found that diff-aware catch generation workflows produce approximately 4 times as many weak catches as hardening workflows. They generate 20 times as many as a baseline that simply attempts to find tests that fail on the diff, and coincidentally pass on its parent revision. Scaling Human-in-the-loop : Based on a manual study of 600 diffs with weak catches, we found that the LLM-as-judge ensemble helped reduce human-in-the loop review load by approximately 70%, allowing us to scale evaluation capacity to approximately 4 times that of manual review alone. Consistent with human labelling : Based on a study of diffs reviewed by humans, and thereby labelled as ‘Good’ (accepted to land into the code base) or ‘Bad’ (not accepted to land), we found statistically significantly more false positive assessments for tests that weakly catch diffs labelled ‘Good’ and significantly more true positive assessments for those labelled ‘Bad’. We also found only modest inter-rater agreement and rank correlation between LLM-based and rule-based assessors. Taken together, these findings suggest that the assessors provide complementary discriminating signals. Severe failures averted : Finally, and most importantly, from 41 reach outs to engineers based on failing test signals from catch, 8 were immediately confirmed to be strong catches, yielding an engineer-experienced true positive rate of 19.5%. Of the 8 strong catches, 4 would have led to serious failures in production. These 4 serious failures were thus averted by Catching JiTTests. We also found that dismissing false positives took only a few minutes and therefore had negligible drag on developer velocity. 2. Baselines for Just in Time Catching A prerequisite for any approach to generate strong catches on a diff is the ability to generate weak catches. In order to assess the degree to which existing technologies can generate weak catches, we used three ‘baseline’ approaches, described below. 2.1. Coincidental regression catches found during ACH hardening deployment We have already deployed mutation-guided LLM-based hardening test generation ( 25 ) into Meta’s continuous integration system, Phabricator ( 24 ) . As a byproduct of this hardening test generation deployment, we also produce tests that fail on