Evaluating Large Language Models for Detecting Architectural Decision Violations

Ruoyu Su 1 , Alexander Bakhtin 1 , Noman Ahmad 1 , Matteo Esposito 1 ,Valentina Lenarduzzi 2 1 , Davide Taibi 2 1 · 2026

Read Evaluating Large Language Models for Detecting Architectural Decision Violations by Ruoyu Su 1 , Alexander Bakhtin 1 , Noman Ahmad 1 , Matteo Esposito 1 ,Valentina Lenarduzzi 2 1 , Davide Taibi 2 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-07609.txt.

Description

Written records of design choices document why a system is built a certain way, but breaks of those rules often go unnoticed for lack of thorough checking. Ruoyu Su and five co-authors test how well language models can spot architectural decision violations in open-source systems.

Their multi-model pipeline screens likely breaks with one model and validates reasoning with three others. It analyzes 980 records across 109 repositories for agreement, accuracy, precision, and recall, plus expert testing. Models agree largely and perform strongly on explicit, code-inferable rules. Software architects get a data-based review of automated architecture conformance checking, with its limits stated plainly.

Catalog record

CREATORRuoyu Su 1 , Alexander Bakhtin 1 , Noman Ahmad 1 , Matteo Esposito 1 ,Valentina Lenarduzzi 2 1 , Davide Taibi 2 1
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-07609

Browse the full library · About the Tower of Babel