Legal Compliance Evaluation of Smart Contracts Generated by Large Language Models
Chanuka Wijayakoon 1 , Hai Dong 1 , H.M.N. Dilum Bandara 2 , Zahir Tari 1 , Anurag Soin 3 · 2025
Read Legal Compliance Evaluation of Smart Contracts Generated by Large Language Models by Chanuka Wijayakoon 1 , Hai Dong 1 , H.M.N. Dilum Bandara 2 , Zahir Tari 1 , Anurag Soin 3 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00943.txt.
Introduction
Chanuka Wijayakoon and co-authors test whether large language models can write smart contracts that stay faithful to the law. Smart contracts automate parts of legal deals in code. Checking their legality by hand is slow and needs rare double expertise.
The team scores compliance by modeling both the legal text and the code as processes and comparing their behavior. Four models each drafted contracts from five legal agreements, making twenty in all. Every model produced code that ran without syntax errors. Legal fidelity, though, varied and fell short in notable ways. Running code is not the same as lawful code. The metrics give the field a way to measure that gap.
Catalog record
| CREATOR | Chanuka Wijayakoon 1 , Hai Dong 1 , H.M.N. Dilum Bandara 2 , Zahir Tari 1 , Anurag Soin 3 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00943
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Legal Compliance Evaluation of Smart Contracts Generated by Large Language Models Thanks: We gratefully acknowledge funding from the Digital Finance CRC which is supported by the Cooperative Research Centres program, an Australian Government initiative. We also thank the Australia and New Zealand Bank for their support and guidance during this research. Chanuka Wijayakoon 1 , Hai Dong 1 , H.M.N. Dilum Bandara 2 , Zahir Tari 1 , Anurag Soin 3 Abstract Smart contracts can implement and automate parts of legal contracts, but ensuring their legal compliance remains challenging. Existing approaches such as formal specification, verification, and model-based development require expertise in both legal and software development domains, as well as extensive manual effort. Given the recent advances of Large Language Models (LLMs) in code generation, we investigate their ability to generate legally compliant smart contracts directly from natural language legal contracts, addressing these challenges. We propose a novel suite of metrics to quantify legal compliance based on modeling both legal and smart contracts as processes and comparing their behaviors. We select four LLMs, generate 20 smart contracts based on five legal contracts, and analyze their legal compliance. We find that while all LLMs generate syntactically correct code, there is significant variance in their legal compliance with larger models generally showing higher levels of compliance. We also evaluate the proposed metrics against properties of software metrics, showing they provide fine-grained distinctions, enable nuanced comparisons, and are applicable across domains for code from any source, LLM or developer. Our results suggest that LLMs can assist in generating starter code for legally compliant smart contracts with strict reviews, and the proposed metrics provide a foundation for automated and self-improving development workflows. I Introduction Blockchains are increasingly used for multi-party business processes [ 1 ] , with users making direct agreements via immutable programs known as smart contracts [ 2 ] . With the tokenized asset market projected to reach US $16.1 trillion by 2030 [ 3 ] , businesses are rapidly developing smart contract-based services on blockchain platforms. To avoid significant legal, financial, and reputational risks, these smart contracts must comply with legal constraints that arise from legal contracts between stakeholders [ 4 , 5 ] and regulatory frameworks in the operating jurisdiction [ 6 ] . Manually implementing legally compliant code is a time-consuming and error-prone process, especially for complex multi-party agreements. With recent advancements in large language models (LLMs) such as GPT-4, there has been a growing interest in using generative models to write code. While LLMs can produce syntactically correct code [ 7 ] , generating smart contracts from natural-language legal agreements faces questions regarding legal compliance. Existing LLM evaluations largely focus on syntactic quality [ 8 ] and basic semantic accuracy [ 9 ] . However,, legal compliance analysis requires complex semantic interpretations and has garnered less attention, while LLMs’ legal reasoning capabilities show high variability [ 10 ] . Furthermore, traditional source-code metrics, such as Cyclomatic Complexity [ 11 ] , fail to capture crucial legal compliance aspects such as rights, obligations, and temporal constraints. Fig. 1: Workflow of Petri-net-based legal and smart contract behavior analysis. To address this gap, we propose a systematic evaluation of the legal compliance of smart contracts generated by leading LLMs, as shown in Fig. 1 . We focus on the alignment between the behaviors captured in a natural-language legal contract and those expressed in the corresponding smart contract, and use it as a proxy to assess LLMs’ capability to generate legally compliant code. To facilitate this comparison, we adopt a contracts-as-processes view [ 12 ] , modeling both legal and smart contracts as Petri Nets (PNs) [ 13 ] to analyze their behaviors. We introduce a novel suite of metrics for legal compliance: 1) fitness , measuring how many behaviors from the legal contract are captured in the smart contract; 2) precision , assessing how many behaviors in the smart contract are relevant to the legal contract; and 3) functional equivalence score (FES), providing a flexible assessment of the alignment between the two contracts. These metrics allow us to quantify the degree of legal compliance for each LLM-generated smart contract, offering a nuanced understanding of how different LLMs perform. As the behavior analysis is independent of how a smart contract is conceived, these metrics can also be applied to any smart contract, from an LLM or a developer. In this paper, we present three primary contributions: 1. A suite of software metrics based on behavior analysis to quantify the legal compliance of source code. 2. A PN-based method to systematically calculate the metrics to determine legal compliance of LLM-generated smart contracts against natural-language legal contracts. 3. A comparative evaluation of smart contracts generated by GPT-4o, Gemini 1.5 Pro, Llama 3.1 70b, and Claude 3.5 Sonnet, highlighting their strengths and weaknesses in achieving legal compliance. The paper is structured as follows: Sec. II outlines background information for our analysis. Sec. III presents the research methodology, with our metrics suite in Sec. IV . Sec. V outlines the experimental setup and presents the results of our smart contract legal compliance evaluation. We discuss our findings and address potential threats to validity in Sec. VI . Sec. VII reviews related work. Concluding remarks and future directions are in Sec. VIII . II Preliminaries We assess smart contract legal compliance by comparing their behaviors against their base legal contracts. Our approach models both contract types as PNs, providing a formal framework for behavioral comparison. This section outlines the theoretical foundation of our methodology. II-A Legal Powers and Obligations Legal contracts are structured around fundamental legal concepts such as rights, duties, powers, and obligations [ 14 ] . These concepts are the foundation for the behaviors that smart contracts need to implement. LLMs, when generating smart contracts, must correctly interpret these legal concepts from the natural-language contract and encode them in code. In this paper, we follow the Symboleo framework [ 15 ] , a formal specification language designed for modeling legal contracts, in choosing legal concepts for analysis: power and obligation. A power is a legal right that enables one party to change the legal positions of another party. An obligation is a duty that requires one party to take specific actions when certain conditions are met. II-B Petri Nets PNs [ 13 ] provide a formal graphical representation to model concurrent systems. They consist of places (ovals) representing resource status or conditions, transitions (rectangles) denoting actions or events, tokens to indicate the current state of the system, and arcs (directed edges) showing the direction of workflows and token movement. Arcs only connect places to transitions or transitions to places. A token in a place indicates an active resource status or a true condition. A transition is enabled if all its incoming places (called the preset ) have tokens. Upon firing, it consumes a token from each incoming place and puts one token each in its outgoing places (called the postset ). An arc with arrow heads on both ends denotes that an associated transition both consumes and produces a token at the connected place. There is also a special inhibitor arcs type that inhibits a transition from firing if the connected place has a