ASSESS: A Semantic and Structural Evaluation Framework for Statement Similarity
Xiaoyang Liu, Tao ZhuZineng DongYuntian LiuQingfeng GuoZhaoxuan LiuYu ChenTao Luo School of Mathematical Sciences, Shanghai Jiao Tong University {xiaoyang.liu, branden2004, stju_dzn, fulcrums, gracegqf,liuzhaoxuan, lcly2462525, luotao41}@sjtu.edu.cn · 2025
Read ASSESS: A Semantic and Structural Evaluation Framework for Statement Similarity by Xiaoyang Liu, Tao ZhuZineng DongYuntian LiuQingfeng GuoZhaoxuan LiuYu ChenTao Luo School of Mathematical Sciences, Shanghai Jiao Tong University {xiaoyang.liu, branden2004, stju_dzn, fulcrums, gracegqf,liuzhaoxuan, lcly2462525, luotao41}@sjtu.edu.cn (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-22246.txt.
Introduction
Xiaoyang Liu and seven co-authors build a better ruler for judging automatic translation of math statements into formal language. Autoformalization is growing, but scoring its output is hard: string comparisons ignore meaning, while proof-based checks give no graded score when a proof fails. Their 2025 framework, ASSESS, accepted at ICLR 2026, scores a formal statement on both meaning and structure.
On those trees, a new metric measures edit distance while allowing semantic transformations, so equivalent rewrites are not punished as errors. To validate it, the team built a benchmark of over a thousand expert-annotated statement pairs from two known collections. Each pair is labeled both for whether one statement can prove the other and for how alike they are structurally. On that benchmark, the new metric beat existing methods on accuracy and on agreement with the human labels.
Catalog record
| CREATOR | Xiaoyang Liu, Tao ZhuZineng DongYuntian LiuQingfeng GuoZhaoxuan LiuYu ChenTao Luo School of Mathematical Sciences, Shanghai Jiao Tong University {xiaoyang.liu, branden2004, stju_dzn, fulcrums, gracegqf,liuzhaoxuan, lcly2462525, luotao41}@sjtu.edu.cn |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-22246
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ASSESS: A Semantic and Structural Evaluation Framework for Statement Similarity Xiaoyang Liu, Tao ZhuZineng DongYuntian LiuQingfeng GuoZhaoxuan LiuYu ChenTao Luo School of Mathematical Sciences, Shanghai Jiao Tong University {xiaoyang.liu, branden2004, stju_dzn, fulcrums, gracegqf,liuzhaoxuan, lcly2462525, luotao41}@sjtu.edu.cn Abstract Statement autoformalization, the automated translation of statements from natural language into formal languages, has seen significant advancements, yet the development of automated evaluation metrics remains limited. Existing metrics for formal statement similarity often fail to balance semantic and structural information. String-based approaches capture syntactic structure but ignore semantic meaning, whereas proof-based methods validate semantic equivalence but disregard structural nuances and, critically, provide no graded similarity score in the event of proof failure. To address these issues, we introduce ASSESS ( A S emantic and S tructural E valuation Framework for S tatement S imilarity ), which comprehensively integrates semantic and structural information to provide a continuous similarity score. Our framework first transforms formal statements into Operator Trees to capture their syntactic structure and then computes a similarity score using our novel TransTED ( Trans formation T ree E dit D istance ) Similarity metric, which enhances traditional Tree Edit Distance by incorporating semantic awareness through transformations. For rigorous validation, we present EPLA ( E valuating P rovability and L ikeness for A utoformalization ), a new benchmark of 524 expert-annotated formal statement pairs derived from miniF2F and ProofNet, with labels for both semantic provability and structural likeness. Experiments on EPLA demonstrate that TransTED Similarity outperforms existing methods, achieving state-of-the-art accuracy and the highest Kappa coefficient. The benchmark, and implementation code will be made public soon. 1 Introduction Formal languages such as Isabelle ( Paulson 1994 ) , HOL Light ( Harrison 1996 ) , Coq ( Barras et al. 1999 ) , and Lean ( De Moura et al. 2015 ; Moura & Ullrich 2021 ) have recently gained prominence within the mathematical community for their capacity to rigorously verify proofs. Nevertheless, formalizing mathematical content is a labor-intensive process that demands substantial time, effort, and a profound familiarity with these specialized languages and their corresponding mathematical libraries, such as Mathlib ( The mathlib Community 2020 ) . Consequently, the task of autoformalization ( Szegedy 2020 ) , defined as translating theorem statements and proofs from natural language into their formal counterparts, has become an active area of research. While autoformalization has rapidly advanced, the methods for evaluating its output have not kept pace. Existing evaluation metrics can be unified under the framework of assigning a similarity score, and they are constrained by a fundamental trade-off between capturing semantic meaning and preserving structural information. String-based metrics such as BLEU ( Papineni et al. 2002 ) rely on surface-level n-gram overlap. This makes them sensitive to inconsequential lexical variations but largely ignores underlying semantic content. Conversely, proof-based approaches ( Li et al. 2024 ; Liu et al. 2025a ) can guarantee semantic equivalence through formal proof. However, this method is brittle; it is fundamentally constrained by the capabilities of the underlying automated theorem prover, leading to a high false negative rate. Furthermore, when a proof cannot be found, these approaches fail to provide any meaningful similarity score. In addition, syntax-based metrics ( Jiang et al. 2023 ) operate at the shallowest level of grammatical compliance, offering no semantic or structural insights. The LLM-as-a-Judge ( Ying et al. 2024a ) approach, while powerful, introduces prohibitive costs and significant reproducibility concerns. These collective shortcomings reveal a critical gap in the field: the need for an automated evaluation metric that robustly combines semantic and structural information for statement similarity in a reproducible and efficient manner. In this work, we propose ASSESS ( A S emantic and S tructural E valuation Framework for S tatement S imilarity ), a novel two-stage framework for evaluating formal statement pairs. The core of ASSESS is its novel metric, TransTED ( Trans formation T ree E dit D istance ) Similarity . It is designed to be sensitive to both semantic and structural nuances while remaining computationally efficient, requiring only CPU resources. In the first stage, ASSESS leverages the Lean Language Server to parse each formal statement pair into its Operator Tree, a representation that captures deeper structural information than raw text. To overcome the semantic limitations of a standard Tree Edit Distance (TED) ( Zhang & Shasha 1989 ) , the second stage augments the TED calculation with a curated set of transformations. This augmentation enables TransTED Similarity to robustly measure statement similarity where purely structural or semantic methods fail. To enable a rigorous evaluation in the absence of a standard benchmark, we introduce EPLA ( E valuating P rovability and L ikeness for A utoformalization ). We constructed this benchmark by first translating informal statements from the miniF2F-test ( Zheng et al. 2022 ) and ProofNet-test ( Azerbayev et al. 2023 ) datasets using the Herald Translator ( Gao et al. 2024 ) and Gemini 2.5 Pro ( Comanici et al. 2025 ) . Successful translations were then paired with their ground-truth counterparts and subsequently annotated by human experts along two axes: semantic provability and structural likeness. The resulting benchmark comprises 524 annotated pairs, drawn from EPLA-miniF2F (373 pairs) and EPLA-ProofNet (151 pairs). To our knowledge, this represents the largest and most comprehensive dataset dedicated to this evaluation task. Experimental results demonstrate that our proposed TransTED Similarity metric consistently outperforms a range of baselines, establishing a new state-of-the-art in both accuracy (78.82% and 70.86%) and the Cohen’s Kappa coefficient (0.46 and 0.40) across both the EPLA-miniF2F and EPLA-ProofNet datasets. Our metric provides a more balanced assessment than brittle proof-based methods, is more efficient and reproducible than LLM-based judges, and exhibits superior robustness compared to n-gram-based metrics. Finally, a detailed ablation study confirms that our novel transformation component is the critical factor behind these performance gains, validating its effectiveness in capturing semantic equivalence. This highlights its success in providing a more balanced and reliable measure of statement similarity. We summarize our main contributions as follows: 1. We propose a novel method that leverages the Lean Language Server to automatically parse formal statements into Operator Trees. This representation enables the direct application of Tree Edit Distance (TED) similarity for evaluating statement similarity. 2. We introduce TransTED Similarity, a semantically-aware metric that extends TED similarity by incorporating transformations directly into the distance calculation. TransTED similarity provides a more holistic similarity score, establishing a new state-of-the-art. 3. We present EPLA, the first large-scale and most comprehensive benchmark for this evaluation task, comprising 524 formal statement pairs meticulously annotated by domain experts, to facilitate rigorous and reproducible evaluation. 2 Related Work Autoformalization. The goal of autoformalization is to translate informal mathematics into formally verified code. Research in this area, particularly for theorem statements, has evolved significantly. While early work often relied on neural machine translation ( Wang et al. 2018 ; Cunningham et al. 2023 ) , the transformative impact of Large Language Models (LLMs) has given rise to three dominant strategies. These include exploring the efficacy of few-shot prompting ( Wu et al. 2022 ; Agrawal et al. 2022 ; Zhou et al. 2024 ) ; improving capabilities by fine-tuning LLMs ( Gao et al. 2024 ; Lu et al. 2024b ; Liu et al. 2025b