Ensuring Reproducibility in Generative AI Systems for General Use Cases: A Framework for Regression Testing and Open Datasets
Masumi Morishige, Ryo Koshihara · 2025
Read Ensuring Reproducibility in Generative AI Systems for General Use Cases: A Framework for Regression Testing and Open Datasets by Masumi Morishige, Ryo Koshihara (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-02854.txt.
Description
Masumi Morishige and Ryo Koshihara published a paper in 2025 on catching quality drift in generative AI systems before users do. A model update or a small prompt edit can quietly change answers, and ordinary testing misses it. The authors propose regression testing, borrowed from software practice, with an open test set to run it on.
Their benchmark, GPR-bench, covers eight task categories with ten scenarios each, in English and Japanese. An automated judge model scores answers for correctness and conciseness. Across three recent model versions, newer models were slightly more correct, but not significantly so, hinting the tests may be too easy to separate them. A concise-writing instruction, by contrast, clearly sharpened brevity at almost no accuracy cost. The set is released openly for community extension.
Catalog record
| CREATOR | Masumi Morishige, Ryo Koshihara |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-02854