Advancing Natural Language Formalization to First Order Logic with Fine-tuned LLMs
Felix Vossel, Till Mossakowski, Björn Gehrke · 2025
Read Advancing Natural Language Formalization to First Order Logic with Fine-tuned LLMs by Felix Vossel, Till Mossakowski, Björn Gehrke (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-22338.txt.
Introduction
Felix Vossel, Till Mossakowski, and Björn Gehrke test how well fine-tuned language models translate plain sentences into first-order logic. That translation underpins knowledge representation and formal methods, where machines must reason over exact statements. Their 2025 study, accepted at a learning and reasoning conference, compares model architectures and training strategies on two known data sets. It introduces scoring measures for exact match, logical equivalence, and how well the predicates, the relation words of logic, line up.
A fine-tuned model from the T5 family, given a list of predicates to use, reached the best accuracy. It beat a leading general model and a strong reasoning model. It also beat a classic symbolic translator. Supplying the predicate list lifted performance by a wide margin. Encoder-decoder models beat larger decoder-only ones at this structured task. And the models generalized to logical arguments they were never trained on.
Catalog record
| CREATOR | Felix Vossel, Till Mossakowski, Björn Gehrke |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-22338
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Advancing Natural Language Formalization to First Order Logic with Fine-tuned LLMs Felix Vossel, Till Mossakowski, Björn Gehrke Abstract Automating the translation of natural language to first-order logic (FOL) is crucial for knowledge representation and formal methods, yet remains challenging. We present a systematic evaluation of fine-tuned LLMs for this task, comparing architectures (encoder-decoder vs. decoder-only) and training strategies. Using the MALLS and Willow datasets, we explore techniques like vocabulary extension, predicate conditioning, and multilingual training, introducing metrics for exact match, logical equivalence, and predicate alignment. Our fine-tuned Flan-T5-XXL achieves 70% accuracy with predicate lists, outperforming GPT-4o and even the DeepSeek-R1-0528 model with CoT reasoning ability as well as symbolic systems like ccg2lambda. Key findings show: (1) predicate availability boosts performance by 15-20%, (2) T5 models surpass larger decoder-only LLMs, and (3) models generalize to unseen logical arguments (FOLIO dataset) without specific training. While structural logic translation proves robust, predicate extraction emerges as the main bottleneck. 1 Introduction Both knowledge representation and formal specification essentially depend on formalisation of knowledge as logical theories. The formalisation process is often a bottleneck, because high-quality formalisation requires manual human effort. As a result, in many ontologies, annotations in natural language are often much richer than the formalised logical theory. Likewise, informal requirement specifications in natural language are often much more extensive than formal specifications. This leads to a limitation of usefulness of formal methods like theorem proving, model finding and model checking, because the involved logical theory is often incomplete and only partially captures the informal natural language specification. In order to address this problem, we aim to automate the translation from natural language to formal logic, more specifically, to first-order logic. While this problem has been solved to a certain extent by different methods, we aim at improving the quality of existing automated translation methods. Our approach uses large language models, which are fine-tuned and evaluated for this specific task, while being simultanesously available as general-purpose LLMs. 2 Related Work Natural Language to First-Order Logic Translation Translating NL to FOL can be used to enable logical reasoning with natural language. Early approaches to this problem relied mainly on symbolic methods. They used rule-based systems with hand-crafted linguistic rules and syntactic patterns, maping natural language structures directly to logical expressions. Pease and Murray developed an early translator from controlled English to logic for ontology-based knowledge representation [ 25 ] . Bos and Markert explored logical inference techniques for recognizing textual entailment [ 2 ] . They used model building from automated reasoning to approximate entailment relationships. Other systems often utilize frameworks like Combinatory Categorial Grammar (CCG) to build compositional semantic representations which can then be transformed to lambda expressions, FOL, or other formal languages. For example, ccg2lambda is a compositional semantics system generating logical semantic representations by combining semantic formulas bottom-up, guided by CCG parse trees [ 19 ] . Barker-Plummer et al. systematically analysed the difficulties of students during a translation task. They identified, among other, the ambiguity of natural language, complex sentence structures, and uninformative automated feedback as the students key challenges in mapping natural language to formal logic [ 1 ] . Cai and Yates addressed the identified scalability by developing techniques for large-scale semantic parsing [ 4 ] . They used schema matching and lexicon extension to improve performance on database-grounded tasks. Martínez-Gómez et al. further refined compositional semantics with the ccg2lambda system [ 19 ] . This system produces logical semantic representations by composing formulas bottom-up from CCG parse trees. More recently, researchers have explored methods beyond purely symbolic systems. Lu et al. demonstrated the use of Dual Reinforcement Learning (DRL) to translate NL to both Propositional and First-Order Logic [ 18 ] . The emergence of large language models (LLMs) offered a new potential solution. However, LLMs often struggle with complex logical reasoning. Several recent works approach this limitations in different ways. LogicLLaMA, a LLaMA-7B model fine-tuned specifically for NL-FOL translation, achieved performance comparable to GPT-4, but at a much lower computational cost [ 30 ] . During their work they created the dataset MALLS, a dataset of 34K high-quality NL-FOL pairs generated using GPT-4. Other frameworks combine LLMs with symbolic reasoners. For example, LogicLM first translates NL to FOL using LLMs and then applies symbolic reasoning [ 24 ] . Experiments showed this combination performed better than using the LLM directly for reasoning. Building on top of that, Lalwani et al. presented NL2FOL [ 12 ] , a framework to translate NL to FOL in multiple steps with few-shot prompting of an LLM. It is designed to detect logical fallacies in natural language statements. Thatikonda et al. focused on fine-tuning LLMs for NL to FOL translation [ 29 ] . They explored various fine-tuning strategies and data augmentation techniques. Their results show that fine-tuning significantly improves performance compared to zero-shot settings. They also found that incrementally translating premises and conclusions of (natural language) logical arguments can improve logical inference checking. Evaluation Metrics For the evaluation of NL to FOL translation tasks, various metrics are used in the literature. A common method is an exact, character-by-character match to evaluate whether a prediction of a model is correct [ 14 , 18 ] . However, this metric is very strict as it does not account for syntactic variations in FOL expressions which do not change the underlying semantics. Other approaches use logical equivalence as a more robust metric. Using automated theorem provers this metric checks whether the predicted FOL expression is logically equivalent to the ground truth expression, even if they are syntactically different [ 3 , 10 ] . However, this approach either requires static signatures ensuring that the same predicates are used in both expressions or a normalization step to align predicate names. Experiments which include NL to FOL translation only as a subtask of logical entailment use the accuracy of the entailment task as a metric [ 29 , 12 ] . Yang et al. defined a new metric called “Logical Equivalence” which is also used in other experiments [ 30 , 16 ] . In contrast to the previous methods, which rate a prediction either “correct” or “incorrect”, this metric assigns a score based on the number of matching propositional valuations. However, it is questionable whether this metric is suitable for FOL, as it does not account for the complexity of quantifiers and a prediction which differs in only one valuation can still be worse than a prediction which differs in many valuations. For example, A\wedge B\wedge C\wedge D is closer to A\wedge B than to \mathit{false} , but differs from the former for \frac{3}{16} of the valuations, but from latter only for \frac{1}{16} . 3 Methodology 3.1 Model Selection For our formalization task, we employ both encoder-decoder and decoder-only large language models (LLMs) with different sizes to enable a systematic comparison across architectures and model sizes. Encoder-decoder models, such as T5-base [ 26 ] , T5-3B [ 26 ] , and Flan-T5-XXL [ 5 ] , are chosen based on their successful application to structured prediction tasks like text-to-text translation and semantic parsing [ 15 ] , which parallel the formality and compositionality of natural language to logic translation. For the decoder-only setting, we