PatentScore: Multi-dimensional Evaluation of LLM-Generated Patent Claims

Yongmin Yoo, Qiongkai Xu, Longbing Cao, {qiongkai.xu, longbing.cao}@mq.edu.au · 2025

Read PatentScore: Multi-dimensional Evaluation of LLM-Generated Patent Claims by Yongmin Yoo, Qiongkai Xu, Longbing Cao, {qiongkai.xu, longbing.cao}@mq.edu.au (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19345.txt.

Introduction

Yongmin Yoo, Qiongkai Xu, and Longbing Cao built a scoring tool for patent claims written by AI in a 2025 paper. Patent claims are strict legal texts where one loose phrase can sink a filing. Normal text scores miss that legal weight, so judging model drafts by them gives a false sense of quality.

Their tool, PatentScore, breaks a claim into parts and scores it on structure, meaning, and legal fit. Checks draw on rules from patent law and craft. On a set of four hundred first claims, its scores matched expert marks more closely than any common text metric. The authors offer it as a base for judging AI drafts in other high stakes fields.

Catalog record

CREATORYongmin Yoo, Qiongkai Xu, Longbing Cao, {qiongkai.xu, longbing.cao}@mq.edu.au
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19345

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PatentScore: Multi-dimensional Evaluation of LLM-Generated Patent Claims Yongmin Yoo, Qiongkai Xu, Longbing Cao, {qiongkai.xu, longbing.cao}@mq.edu.au Abstract High-stakes texts such as patent claims, medical records, and technical reports are structurally complex and demand a high degree of reliability and precision. While large language models (LLMs) have recently been applied to automate their generation in high-stakes domains, reliably evaluating such outputs remains a major challenge. Conventional natural language generation (NLG) metrics are effective for generic documents but fail to capture the structural and legal characteristics essential to evaluating complex high-stakes documents. To address this gap, we propose PatentScore, a multi-dimensional evaluation framework specifically designed for one of the most intricate and rigorous domains, patent claims. PatentScore integrates hierarchical decomposition of claim elements, validation patterns grounded in legal and technical standards, and scoring across structural, semantic, and legal dimensions. In experiments on our dataset which consists of 400 Claim1, PatentScore achieved the highest correlation with expert annotations ( r = 0.819 r=0.819 ), significantly outperforming widely used NLG metrics. This work establishes a new standard for evaluating LLM-generated patent claims, providing a solid foundation for research on patent generation and validation. 1 Introduction Large language models (LLMs) have demonstrated impressive capabilities in automated text generation across a wide range of natural language processing tasks. However, existing evaluation frameworks remain insufficient for rigorously and systematically assessing their outputs in high-stakes domains. These domains, such as patent claims, legal contracts, and technical reports, involve deeply hierarchical and interdependent structures that cannot be accurately evaluated using surface-level metrics such as sentence-level fluency or shallow contextual similarity. Current evaluation metrics often fail to capture critical domain-specific nuances, including legal constraints, structural dependencies, technical precision, and implicit reasoning, which significantly limit their effectiveness in these contexts. This limitation is especially consequential because it poses risks to both legal validity and practical utility of the generated documents, thereby hindering their adoption in real-world, risk-sensitive applications ( Guha et al., 2023 ) . Figure 1: A comparison of PatentScore and standard NLG metrics in evaluating structural and legal accuracy of patent claims. Among high-stakes legal and technical documents, patent claims pose particularly significant challenges for evaluation. Because they function as both technical descriptions and legally binding texts, patent claims require precise assessment of structural elements such as antecedent consistency and claim dependency. Existing natural language generation (NLG) evaluation methods struggle in domains where both linguistic precision and legal accountability are critical. Patent claims exemplify these characteristics, as they define the scope of rights granted to an invention. The first claim, in particular, plays a central role in shaping the structure and coverage of the entire claim set. This pivotal role entails substantial socio-economic implications, as it determines the legal and commercial value of the invention ( Merges, 1997 ; Bessen and Meurer, 2009 ; Surden, 2019 ) . With the rapid advancement of LLMs, recent studies have explored their potential to generate and evaluate precision-critical legal texts. However, patent claims remain a particularly challenging domain due to their intricate structural and legal constraints ( Aristodemou and Tietze, 2018 ; Lee and Hsiang, 2020 ) . Traditional NLG metrics, such as BLEU ( Papineni et al., 2002 ) , ROUGE ( Lin, 2004 ) , and BERTScore ( Zhang et al., 2020 ) , primarily focus on lexical overlap or contextual similarity, failing to address the structural and legal nuances required for patent claim evaluation. More recent LLM-based evaluation approaches, such as LLM-as-a-Judge ( Zheng et al., 2023 ) , rely on general-purpose models to assess fluency and semantic coherence. However, they lack the domain-specific knowledge required to evaluate structural constraints, including claim dependency and legal clarity, that are critical to patent claims. These shortcomings limit the applicability of such metrics in high-stakes legal contexts where precision is paramount ( Zuo et al., 2024 ) . For instance, inconsistent antecedent references can render a patent legally ambiguous, while punctuation errors may distort its scope of protection ( Merges, 1997 ) . To address these limitations, we propose PatentScore, a multi-dimensional evaluation framework for patent claims. This framework evaluates structural and legal elements such as antecedent consistency, claim dependency, and legal clarity. Our approach decomposes claims into constituent parts and applies targeted metrics, enabling reliable and consistent assessment of LLM-generated claims. This systematic evaluation significantly enhances the quality of claim evaluation compared to existing metrics. Our objective is to develop a domain-specific metric for patent claims while establishing a generalizable, structure-aware evaluation framework applicable to high-precision documents, such as contract clauses, policy reports, and regulatory compliance in clinical documentation. The key contributions of PatentScore are summarized as follows: • Claim-Structured Evaluation Framework. We introduce an architecture that decomposes a claim into structural components and quantifies each, covering claim structure, element linkage, antecedent basis, and claim dependency per official examination guidelines. • Expert-Aligned Validation. We compare PatentScore’s evaluation of LLM-generated claims against expert ratings, achieving a strong correlation ( r=0.819 ) and outperforming standard NLG metrics in capturing structural and legal precision. • Open Evaluation Benchmark. We release 400 LLM-generated claims with expert annotations, creating the first public benchmark for evaluating LLM-generated patent texts. Although our experiments focus on patent claims, PatentScore is model-agnostic. Its structural and legal principles extend to other high-precision texts such as contract clauses, policy reports, clinical documentation, and regulatory filings. By providing a robust framework for evaluating structured texts, our work lays the foundation for establishing reliable evaluation protocols essential for the responsible use of LLMs in high-stakes domains. 2 Background and Related Work 2.1 Specifications in Patent Claims Claim 1 (Example) 1. An extendible trailer tongue comprising: a first tubular member; a second tubular member telescopically received within the first tubular member; and a locking mechanism to secure the second tubular member in a selected position relative to the first tubular member. Table 1: A Brief summary example of Claim 1 from the patent titled “Multiple component headgear system” ( Siprut, 2000 ) . Patents are specialized legal-technical documents designed to protect intellectual property (IP) rights. Among their components, claims play a central role, as they define the precise legal boundaries of protection granted to an invention. In particular, Claim 1 , which is the first claim, typically serves as the cornerstone of the claim set. It establishes the broadest scope of protection and strongly influences both the validity and enforceability of the patent as a whole ( World Intellectual Property Organization, 2022 ; Merges, 1997 ) . The breadth of Claim 1 not only constrains subsequent dependent claims but also directly affects the legal and commercial value of the invention. Unlike general technical documents, patent claims must strictly adhere to structural, linguistic, and legal requirements. They follow a standardized format mandated by patent offices, employ precise domain-specific terminology, and are drafted with enforceability in mind. Table 1 illustrates an example of a well-structured Claim 1, which demonstrates these key requirements: • Structural format: Compliance with strict grammatical and formatting conventions, including numbered clauses, transitional phrases (e.g., “comprising”), and hierarchical listing of elements. Deviations can render a claim unclear or legally unenforceable. • Technical clarity: Precise definition of technical element and its relationships, ensuring that the scope of the invention is unambiguous to examiners, practitioners, and courts.

Read the complete text