(How) Do Large Language Models Understand High-Level Message Sequence Charts?

  • Authors: Mohammad Reza Mousavi
  • Contributing partner: King's College London
  • Published in: Leveraging Applications of Formal Methods, Verification and Validation. Rigorous Engineering of Collective Adaptive Systems: 13th International Symposium, ISoLA 2026, Kos, Greece, October 24-28, 2026, Proceedings, Part I (Lecture Notes in Computer Science), Springer, pp. 126-146
  • Published: Oct 2026
  • Type: Book chapter
  • DOI: 10.1007/978-3-032-40108-3_8

Synopsis

Large language models are now used across software development, but it is unclear whether they handle design artefacts in line with what those artefacts formally mean, and this has had little attention for architectural specifications. This paper tests the question on High-Level Message Sequence Charts, visual models with a rigorous formal semantics that underpin sequence diagrams in UML. Three models, Gemini-3, GPT-5.4 and Qwen-3.6, were set 129 semantic tasks: questions about events and their ordering, abstractions and compositions that should preserve meaning, and calculating sets of traces and equivalent labelled transition systems. Overall accuracy was modest, at about 52%. The models handled the basic concepts well, at about 88%, but reached only about 36% on abstraction and composition and about 42% on traces and transition systems. All three struggled with co-regions and explicit causal dependencies and never used them when transforming charts. The results argue for caution before relying on language models for tasks that depend on the precise meaning of design models.

Cite this work

Mousavi, M. R. (2026). (How) Do Large Language Models Understand High-Level Message Sequence Charts? Leveraging Applications of Formal Methods, Verification and Validation. Rigorous Engineering of Collective Adaptive Systems, 126–146. https://doi.org/10.1007/978-3-032-40108-3_8