• 01 October, 2026
  • GreenCode

New paper: (How) Do Large Language Models Understand High-Level Message Sequence Charts?

Large language models are now used across software development, but it is unclear whether they handle design artefacts in line with what those artefacts formally mean, and this has had little attention for architectural specifications. This paper tests the question on High-Level Message Sequence Charts, visual models with a rigorous formal semantics that underpin sequence diagrams in UML. Three models, Gemini-3, GPT-5.4 and Qwen-3.6, were set 129 semantic tasks: questions about events and their ordering, abstractions and compositions that should preserve meaning, and calculating sets of traces and equivalent labelled transition systems. Overall accuracy was modest, at about 52%. The models handled the basic concepts well, at about 88%, but reached only about 36% on abstraction and composition and about 42% on traces and transition systems. All three struggled with co-regions and explicit causal dependencies and never used them when transforming charts. The results argue for caution before relying on language models for tasks that depend on the precise meaning of design models.

Authors: Mohammad Reza Mousavi

Contributing partner: King's College London

Publication page and citation View at the publisher