Evaluating Large Language Models for Detecting Architectural Decision Violations
Ruoyu Su 1 , Alexander Bakhtin 1 , Noman Ahmad 1 , Matteo Esposito 1 ,Valentina Lenarduzzi 2 1 , Davide Taibi 2 1 · 2026
Read Evaluating Large Language Models for Detecting Architectural Decision Violations by Ruoyu Su 1 , Alexander Bakhtin 1 , Noman Ahmad 1 , Matteo Esposito 1 ,Valentina Lenarduzzi 2 1 , Davide Taibi 2 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-07609.txt.
Description
Written records of design choices document why a system is built a certain way, but breaks of those rules often go unnoticed for lack of thorough checking. Ruoyu Su and five co-authors test how well language models can spot architectural decision violations in open-source systems.
Their multi-model pipeline screens likely breaks with one model and validates reasoning with three others. It analyzes 980 records across 109 repositories for agreement, accuracy, precision, and recall, plus expert testing. Models agree largely and perform strongly on explicit, code-inferable rules. Software architects get a data-based review of automated architecture conformance checking, with its limits stated plainly.
Catalog record
| CREATOR | Ruoyu Su 1 , Alexander Bakhtin 1 , Noman Ahmad 1 , Matteo Esposito 1 ,Valentina Lenarduzzi 2 1 , Davide Taibi 2 1 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-07609