LLM-Based Test Case Generation in DBMS through Monte Carlo Tree Search Conference: 2026 IEEE/ACM 48th International Conference on Software Engineering; April 12–18, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE-Companion ’26), April 12–18, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3774748.3793625 ISBN: 979-8-4007-2296-7/2026/04 ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Artificial intelligence CCS: Software and its engineering Software post-development issues

Yujia Chen, Yingli Zhou, Fangyuan Zhang, Cuiyun Gao · 2026

Read LLM-Based Test Case Generation in DBMS through Monte Carlo Tree Search Conference: 2026 IEEE/ACM 48th International Conference on Software Engineering; April 12–18, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE-Companion ’26), April 12–18, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3774748.3793625 ISBN: 979-8-4007-2296-7/2026/04 ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Artificial intelligence CCS: Software and its engineering Software post-development issues by Yujia Chen, Yingli Zhou, Fangyuan Zhang, Cuiyun Gao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21530.txt.

Introduction

Chen and co-authors generate test cases for database systems with help from language models and Monte Carlo tree search. Fuzzers work well but need heavy manual tuning for each proprietary SQL dialect. Lightweight models, common inside companies, often write invalid queries and quickly run out of variety. Their MIST framework builds a feature tree and uses error feedback to guide valid, diverse query synthesis. Tree search then mutates seed queries toward deeper code paths. Coverage rose sharply over the best baseline on three systems.

Catalog record

CREATORYujia Chen, Yingli Zhou, Fangyuan Zhang, Cuiyun Gao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51737
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.21530

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLM-Based Test Case Generation in DBMS through Monte Carlo Tree Search Conference: 2026 IEEE/ACM 48th International Conference on Software Engineering; April 12–18, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE-Companion ’26), April 12–18, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3774748.3793625 ISBN: 979-8-4007-2296-7/2026/04 ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Artificial intelligence CCS: Software and its engineering Software post-development issues Yujia Chen, Yingli Zhou, Fangyuan Zhang, Cuiyun Gao Abstract . Database Management Systems (DBMSs) are fundamental infrastructure for modern data-driven applications, where thorough testing with high-quality SQL test cases is essential for ensuring system reliability. Traditional approaches such as fuzzing can be effective for specific DBMSs, but adapting them to different proprietary dialects requires substantial manual effort. Large Language Models (LLMs) present promising opportunities for automated SQL test generation, but face critical challenges in industrial environments. First, lightweight models are widely used in organizations due to security and privacy constraints, but they struggle to generate syntactically valid queries for proprietary SQL dialects. Second, LLM-generated queries are often semantically similar and exercise only shallow execution paths, thereby quickly reaching a coverage plateau. To address these challenges, we propose MIST , an LL M -based test case generat I on framework for DBM S through Monte Carlo T ree search. MIST consists of two stages: Feature-Guided Error-Driven Test Case Synthetization , which constructs a hierarchical feature tree and uses error feedback to guide LLM generation, aiming to produce syntactically valid and semantically diverse queries for different DBMS dialects, and Monte Carlo Tree Search-Based Test Case Mutation , which jointly optimizes seed query selection and mutation rule application guided by coverage feedback, aiming at boosting code coverage by exploring deeper execution paths. Experiments on three widely-used DBMSs with four lightweight LLMs show that MIST achieves average improvements of 43.3% in line coverage, 32.3% in function coverage, and 46.4% in branch coverage compared to the baseline approach with the highest line coverage of 69.3% in the Optimizer module. † † cc-license: by 1. Introduction Figure 1 . An illustrative example of using Qwen2.5-7B to generate a SQL test case for DBMS. Database Management Systems (DBMSs) are fundamental infrastructure for modern data-driven society, underpinning applications across finance, healthcare and e-commerce ( 36 ; 29 ; 25 ) . Ensuring the correctness and robustness of DBMSs is vital, as even minor faults can lead to severe consequences, including data corruption, service outages, or security breaches ( 21 ; 3 ; 26 ) . An essential step toward assuring DBMS reliability is thorough testing using high-quality SQL test cases, which help uncover subtle defects before deployment and prevent them from affecting production environments ( 48 ; 2 ) . Traditional approaches for generating test cases, such as BuzzBee ( 46 ) , SQLsmith ( 34 ) , SQLancer ( 32 ) , can be effective for a specific DBMS; however, there exist many DBMSs designed for different application scenarios, such as DuckDB ( 29 ) for analytical workloads, PostgreSQL ( 36 ) for general-purpose OLTP systems, and SQLite ( 24 ) for lightweight embedded environments. Adapting these methods to different DBMSs requires substantial manual effort, such as crafting grammar rules and maintaining specific operators, which severely limits their scalability ( 49 ; 50 ; 48 ) . Recent advances in Large Language Models (LLMs) present promising opportunities for DBMS test case generation. First, LLMs have already demonstrated success in related data management tasks, including text-to-SQL ( 4 ; 17 ; 18 ) , data cleaning ( 19 ; 22 ; 5 ; 27 ) , knob tuning ( 16 ; 8 ) , and DBMS diagnosis ( 52 ; 47 ; 35 ) . Second, LLMs are applied in testing across multiple domains, such as general software testing ( 39 ; 43 ) , compiler testing ( 10 ; 42 ) , and embedded testing ( 1 ) . As illustrated in Figure 1 , LLMs can directly generate SQL test cases by leveraging their understanding of SQL syntax and semantics acquired during pre-training. Despite these encouraging developments, deploying LLMs for industrial-scale DBMS testing still faces two critical challenges: • Challenge 1: Limited model adaptability in proprietary and resource-constrained industrial DBMS environments. Industrial DBMSs often have proprietary SQL dialects with unique syntax and features. At the same time, security and privacy requirements restrict organizations to locally deployed lightweight LLMs (typically 70B parameters or smaller). The limited model capacity results in generating overly simple queries with insufficient testing coverage (see Figure 2 (a)). Moreover, these models lack sufficient training on proprietary SQL variants, often producing queries that violate dialect-specific rules (see Figure 2 (b)). As a result, adapting lightweight LLMs to proprietary industrial DBMSs remains challenging. • Challenge 2: Limited exploration of deeper execution paths in test generation. Effective DBMS testing requires queries that cover meaningful execution paths beyond syntactic correctness. However, simply generating a large number of test cases does not translate into high coverage, as many queries are semantically similar and exercise only shallow execution paths. As demonstrated in a prior study ( 40 ) , the generated test cases may improve coverage initially, but the coverage quickly plateaus as new inputs rarely explore deeper logic. Establishing an effective mechanism to guide test generation toward unexplored code regions remains a critical challenge. ⬇ CREATE TABLE products ( id INTEGER , name TEXT , price REAL ); INSERT INTO products VALUES (1, ’Laptop’ , 999.99), (2, ’Phone’ , 499.99); SELECT name , price FROM products WHERE price > 10; (a) Simple query with limited testing coverage. ⬇ CREATE TABLE employees ( id INTEGER , details STRUCT<age: INTEGER> ); INSERT INTO employees VALUES (1, 5); SELECT * FROM products WHERE name REGEXP_MATCHES ( ’^A’ ) CROSS JOIN cities c ON c . id =1; (b) Query violating syntax rules on DuckDB (errors highlighted in red ). Figure 2 . Illustrating challenges in LLM-based DBMS test case generation with Qwen2.5-7B. Our work. To address these challenges, we propose MIST , an LL M -based test case generat I on framework for DBM S through Monte Carlo T ree search. MIST consists of two complementary stages: 1) Feature-Guided Error-Driven Test Case Synthetization. This stage constructs a feature tree from official documentation and hierarchically samples feature combinations to guide LLM generation. Execution errors are captured and fed back to subsequent prompts, improving syntactic validity and semantic diversity of synthesized test cases. 2) Monte Carlo Tree Search-Based Test Case Mutation. After coverage growth plateaus, this stage employs MCTS to iteratively select seed queries and apply mutation rules guided by coverage feedback from test execution. With elaborate mutation rules covering schema, data, and query operations, this stage systematically explores deeper execution paths to boost code coverage. We evaluate MIST on three widely-used open-source DBMSs (DuckDB, PostgreSQL, and SQLite) with four LLMs of different parameter scales (Qwen2.5-7B ( 28 ) , Llama3.1-8B ( 9 ) , Qwen2.5-14B ( 28 ) , and Qwen2.5-32B ( 28 ) ). Experimental results show that MIST improves the baseline approach by 23.9% \sim 69.17% on DuckDB, 18.5% \sim 54.1% on PostgreSQL, and 18.5% \sim 92.9% on SQLite in terms of average code coverage. Moreover, MIST achieves the highest line coverage in the Optimizer module, reaching

Read the complete text