Teaching LLMs Program Semantics via Symbolic Execution Traces
Jonas Bayer, Stefan Zetzsche, Olivier Bouissou, Remi Delmas, Michael Tautschnig, Soonho Kong · 2026
Read Teaching LLMs Program Semantics via Symbolic Execution Traces by Jonas Bayer, Stefan Zetzsche, Olivier Bouissou, Remi Delmas, Michael Tautschnig, Soonho Kong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06184.txt.
Introduction
Jonas Bayer and five co-authors teach language models program semantics using symbolic execution traces. Models often confirm that code properties hold, yet they miss violations, and that weakness grows with program length. The team first build an evaluation of 500 C verification tasks across five property types, from memory safety to data races, and test fourteen models.
Training data comes from running a symbolic execution engine on generic open source C code and collecting its traces. About 3,000 bug traces, paired with chain of thought reasoning at test time, raised violation detection by over seventeen points. Neither ingredient alone helped much, but together they reinforced each other strongly. The trained eight billion parameter model beat a four times larger sibling on violation detection and gained across all property types.
Catalog record
| CREATOR | Jonas Bayer, Stefan Zetzsche, Olivier Bouissou, Remi Delmas, Michael Tautschnig, Soonho Kong |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 81 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.06184
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Teaching LLMs Program Semantics via Symbolic Execution Traces Jonas Bayer, Stefan Zetzsche, Olivier Bouissou, Remi Delmas, Michael Tautschnig, Soonho Kong Abstract We introduce an evaluation framework of 500 C verification tasks across five property types (memory safety, overflow, termination, reachability, data races) built on SV-COMP 2025, and evaluate 14 models across six families. We find that high overall accuracy masks a critical weakness: while most models reliably confirm properties hold, violation detection varies widely and degrades sharply with program length. To close this gap, we train on formal verification artifacts: running the Soteria symbolic execution engine on generic open-source C code and using the resulting traces for continued pretraining of Qwen3-8B. Just ∼ {\sim} 3,000 bug traces combined with chain-of-thought reasoning at inference time improve violation detection by over 17 percentage points, producing one of the most balanced accuracy profiles among evaluated models. On violation detection, the trained 8B model outperforms the 4 × \times larger Qwen3-32B without thinking and approaches it in overall accuracy. The interaction between trace training and chain-of-thought is superadditive: neither alone provides meaningful gains, but their combination does. Improvements transfer across all five property types, including ones the training traces do not target. Our 28 configurations confirm the gains stem from trace semantics, not code volume, and that trace curation and format matter. 1 Introduction How well do LLMs understand low-level code? For higher-level languages, code understanding evaluations are approaching saturation [ 37 ] , but C requires reasoning about memory safety, pointer arithmetic, and undefined behavior, where models remain less reliable [ 29 ] . Yet there is no established evaluation framework that isolates this kind of semantic reasoning for C. We introduce an evaluation framework built on SV-COMP 2025 [ 6 ] , the Software Verification Competition. Given a C program and a property (memory safety, absence of overflows, termination, reachability, or absence of data races), models must determine whether the property holds or is violated, with ground truth established by formal verification tools. We evaluate 14 models across six families and find that while most models achieve over 90% accuracy on confirming properties hold, violation detection is far weaker: four of fourteen models correctly identify fewer than 50% of actual violations, and accuracy drops sharply with program length, with one model falling below 10% on programs of just 100–200 lines (Figure 2 ). Notably, this does not correlate straightforwardly with model size or code specialization; for instance, GPT-OSS-20B detects violations more reliably than Mistral Large 3 675B and Qwen3-Coder-480B, suggesting that training data and objectives play a larger role than scale. A failure analysis (Appendix B.5 ) reveals that models fail by missing triggering edge cases, making incorrect logical or arithmetic reasoning steps, or identifying the violation but incorrectly concluding it is benign. To improve violation detection, we explore using formal verification artifacts as training data. Prior work combining LLMs with formal methods has largely focused on inference time; the training-time direction is less explored. Existing efforts have targeted theorem proving with Lean proofs [ 27 , 30 ] and training on concrete Python execution traces [ 13 ] . We extend this idea to C, where a mature ecosystem of symbolic execution tools exists, yet their artifacts remain untapped for LLM training. We focus on symbolic execution, which reasons about all possible inputs simultaneously rather than sampling concrete values. Symbolic execution artifacts capture program semantics in a form well-suited for training: they trace execution paths, track memory states, and reveal the precise conditions under which properties are violated. We use Soteria [ 4 ] , an open-source symbolic execution engine that is fast, produces detailed traces, and supports both C and Rust. Starting from the GitHub CodeParrot dataset [ 12 ] , we filter to 1 million self-contained C files and run Soteria on each, yielding approximately 34,000 usable traces. The reduction stems from timeout and parser limitations in Soteria; we expect the yield to improve as the tooling matures. We conduct extensive ablation studies on Qwen3-8B to understand how different aspects of the training data affect performance. Our main finding is that filtering to manifest bug traces (where Soteria proves a violation occurs on all feasible inputs), combined with chain-of-thought reasoning at inference time, yields a 17.9 percentage point improvement on detecting property violations. The reported holds accuracy drops from 90.7% to 78.0%, but this is driven entirely by responses that exhaust the output token budget; on parseable responses, holds accuracy is preserved (90.8% vs. 91.2% baseline) while violated accuracy improves by +26 pp. This gain depends on careful curation: training on all Soteria traces degrades performance, and training on the underlying C files alone also hurts, confirming that improvement derives from the semantic information in Soteria’s traces rather than simply increased code exposure. Importantly, the training data is entirely independent of our benchmark, drawn from arbitrary open-source C code, demonstrating that improvements transfer to unseen verification tasks. Claude Opus 4.7 Kimi K2.5 DeepSeek V3.2 GPT-OSS 120B Claude Son. 4.5 DeepSeek R1 GPT-OSS 20B Qwen3 Coder 480B Qwen3 32B Mistral Large 3 675B Qwen3 8B Devstral 2 123B Qwen3 Coder 30B Ministral 3 8B 0 50 100 95.1 91.9 91.8 91.3 89.6 89.5 84.4 84.2 76.2 74.0 70.0 69.7 68.1 67.8 Accuracy (%) Property Holds Overall Property Violated Figure 1 : Baseline accuracy on our benchmark, sorted by overall performance. Numbers above each bar show overall accuracy (%). All models detect property holds more reliably than violations, with the gap widening dramatically for the Mistral and Qwen families regardless of scale. Figure 2 shows that violation detection degrades sharply already within the 100–200 LOC range. Our contributions are: • An evaluation framework of 500 verification tasks across five property types, with ground truth from SV-COMP 2025. A baseline evaluation of 14 models reveals that violation detection varies widely — from 92% for the best model down to below 40% — a deficit often masked by strong performance on holds cases, which inflates overall accuracy. • A scalable, fully automated, annotation-free training pipeline: run Soteria on generic open-source C code, filter to manifest bug traces, and continue pretraining. Just 3,208 traces and \sim 10 minutes of training improve Qwen3-8B violation detection by +17.9 pp while also improving overall accuracy. Although the training traces primarily target memory safety and overflows, violation detection improves across all five evaluation properties, suggesting the model acquires general verification reasoning. • Extensive ablations (28 configurations) showing that gains are driven by the semantic content of symbolic execution traces, not code volume, and that trace format, curation, and thinking mode interact non-obviously: CPT (continued pretraining) and thinking are individually weak but superadditive in combination (+17.9 pp vs. +7.3 pp and - 1.4 pp alone), and preserving the original structured trace format is critical for chain-of-thought to succeed. 2 Related Work LLMs for Code Understanding. Benchmarks for semantic code understanding remain limited. CRUXEval [ 18 ] evaluates input-output prediction for Python but not low-level languages; CoRe [ 37 ] tests data and control dependencies across Python, C, and Java but is approaching saturation; LoCoBench [ 29 ] evaluates long-context understanding across 10 languages with lower C/C++ performance. In security, recent work observes that models struggle more with detecting vulnerabilities