(How) Do Large Language Models Understand High-Level Message Sequence Charts?

Mohammad Reza Mousavi · 2026

Read (How) Do Large Language Models Understand High-Level Message Sequence Charts? by Mohammad Reza Mousavi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13773.txt.

Introduction

Mohammad Reza Mousavi asks whether large word models grasp the formal meaning of High-Level Message Order Charts. He tests three models on 129 meaning tasks from querying basic constructs to meaning-keeping lifts and joins. Results show only modest grasp at about 52 percent overall hits. Basic-concept scores are strong near 88 percent, but lifts and joins scores are weak near 36 percent, and traces and LTS scores near 42 percent. All three models struggle with co-region and plain causal links, never using them in meaning-keeping changes.

Catalog record

CREATORMohammad Reza Mousavi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39763
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.13773

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

(How) Do Large Language Models Understand High-Level Message Sequence Charts? Mohammad Reza Mousavi Abstract Large Language Models (LLMs) are being employed widely to automate tasks across the software development life-cycle. It is, however, unclear whether these tasks are performed consistently with respect to the semantics of the artefacts being handled. This question is particularly under-researched concerning architectural design specification. In this paper, we address this question for High-level Message Sequence Charts (HMSCs). These are visual models with a rigorous formal semantics that have been used for various purposes, including as a foundation for Sequence Diagrams in the Unified Modelling Language (UML). We examine whether LLMs “understand” the semantics of HMSCs by examining three LLMs (Gemini-3, GPT-5.4, and Qwen-3.6) on how they perform 129 semantic tasks ranging from querying basic semantic constructs in HMSCs (i.e., events and their ordering) to semantic-preserving abstractions and compositions, and calculating the set of traces and trace-equivalent labelled transition systems. The results show that LLMs only have a modest understanding of the formal semantics of HMSCs ( ∼ \sim 52% overall accuracy), with great variability across different semantic concepts: while LLMs seem to understand the basic semantic concepts of MSCs ( ∼ \sim 88% accuracy), they struggle with semantic reasoning in tasks involving abstraction and composition ( ∼ \sim 36% accuracy) and traces and LTSs ( ∼ \sim 42% accuracy). In particular, all three LLMs struggle with the notions of co-region and explicit causal dependencies and never employed them in semantic-preserving transformations. This paper is dedicated to Sjouke Mauw and Michel Reniers. To Sjouke Mauw, on the occasion of his retirement. To Sjouke Mauw and Michel Reniers, for their contributions to the formal semantics of Message Sequence Charts; and To Michel Reniers, for teaching me formal semantics…. 1 Introduction 1.1 Motivation High-level Message Sequence Charts (HMSCs) [ 1 ] are among the pioneering models for software and system specification, which are furnished with a formal semantics. They have been widely used in practice, standardised, and have also been integrated into the Unified Modelling Language [ 2 ] in the form of sequence diagrams. 1.2 Problem Definition Understanding architectural specification is among the challenging usecases for Large Language Models (LLMs) [ 5 ] . By “understanding” we do not mean to anthropomorphise LLMs [ 13 ] ; rather, we are referring to their observable behaviour: demonstrating outputs that are consistent with a sound model of the semantics of HMSCs. It is unclear whether LLMs respect the semantics of architectural specification languages, even with respect to their most fundamental semantics construct. 1.3 Approach In this study, our aim was to quantify the LLMs’ understanding of HMSCs. To this end, we identify the key semantic concepts in the formal semantics of HMSCs; formulate and carry out tasks that examine their semantic understanding; and analyse the performance of LLMs and provide a quantification of the understanding of various semantic concepts. Key semantic concepts. The formal semantic concepts are drawn from the formalisation of HMSCs, based on the ITU Recommendation Z.120 Annex B: Algebraic Semantics of Message Sequence Charts [ 1 ] . The identified semantics concepts are: causal ordering of events, abstraction, composition, traces, and (trace-equivalent) labelled transition systems. We expand on each of these concepts and identify patterns that exemplify them in the subsequent explanation, generation, and manipulation tasks. 1.4 Results Our results indicate that the three LLMs evaluated in our paper, all have a modest understanding of the formal semantics of HMSCs ( \sim 52% overall accuracy), with great variability across different semantic concepts: they all have a precise model of the basic semantics constructs of (H)MSCs such as their comprising events and event orderings ( \sim 88% accuracy). However, they struggle with abstraction, composition ( \sim 36% accuracy), and higher-level reasoning about behavioural semantics ( \sim 42% accuracy). They all seem to lack a sound notion of co-region (a region with no ordering of events) and explicit causal orders among events. For example, they never use co-regions when composing (H)MSCs horizontally (in parallel), leading to introducing fictitious orders in such compositions. Also, when abstracting from events, they cannot reinstate the ordering implied by the abstracted events, leading to fewer orders than the original (H)MSC. Another interesting observation is that in the few cases of successful higher-level reasoning, they generate Python code for the corresponding reasoning task and execute it to obtain a sound result. This indicates that potentially reliable results can be obtained by interfacing LLMs with other tools and encoding the reasoning tasks into the input of such external tools. The details of experiments, results, and analysis are publicly available through the replication package: https://smrmousavi.github.io/sjouke-fest/ . 1.5 Organisation The remainder of this paper is organised as follows. We review the related work to position our work with respect to the literature on understanding code and architectural diagrams in the context of Large Language Models in Section 2 . Then we recall some basic concepts regarding the syntax and semantics of (H)MSCs in Section 3 . In the same section, we identify the key semantic concepts that we would like to examine in our experiments. This leads to a formulation of our research questions and the design of experiments to answer them in Section 4 . We report the results of our experiments and answer the formulated research questions in Section 5 . Finally, we reflect on these answers and on the threats to the validity and the generalisability of the results in Section 6 . 2 Related Work In this section, we review the related lines work on measuring semantic understanding in LLMs for code and for architectural specifications. Understanding Program Semantics. Although understanding code semantics may deal with more fine-grained tasks than the architectural tasks studied in this paper, many of the semantic concepts examined in this paper are still relevant at code-level (operational) semantics, in particular for reasoning about concurrency. Ma et al. [ 9 ] analyse the LLM capabilities in code understanding at three levels: syntax, static semantics, and dynamic semantics. For understanding dynamic semantics, they report an accuracy of ca. 60% using zero-shot (and slightly lower using few-shot) prompting. Laneve et al. [ 8 ] analyse program semantic understanding among seven different LLMs. They do so by generating tasks about program equivalence and analysing the correctness of the outcome. They report a mean accuracy of 59% in zero-shot prompting and 71% after providing contextual information about the perturbation in the programs. They conclude that there is a significant gap in the semantic reasoning capabilities of the analysed LLMs. Similarly, Wei et al. [ 18 ] analyse the capabilities of 19 LLMs to reason about the programme equivalence and report a mean precision of 64%. Hooda et al. [ 7 ] use counter-factual analysis to analyse LLM program understanding: they use four different Program Concept Predicates to characterise four classes of semantic equivalence in programs. They use these concepts to evaluate the performance of LLMs in code completion and program repair tasks. Their analysis shows significant gaps in understanding program semantics that influence downstream task precision by up to 33%. All of these approaches address the issue of semantic understanding, and point out significant gaps in understanding code semantics. Our methodology differs from

Read the complete text