Towards Corpus-Grounded Agentic LLMs for Multilingual Grammatical Analysis
2025
Read Towards Corpus-Grounded Agentic LLMs for Multilingual Grammatical Analysis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-00214.txt.
Introduction
Grammar research increasingly runs on large annotated text collections, yet analyzing them still takes heavy technical effort. A 2025 study examines how agentic large language models could streamline that work across languages. The models reason over annotated corpora instead of single example sentences.
The goal is interpretable, data-grounded answers to linguistic questions, produced with less manual setup. Corpus-grounded agents could help linguists test claims about grammar against real usage at scale. The study speaks to a wider shift in the humanities, where language models are becoming research assistants rather than just text generators. Multilingual coverage matters because grammar traditions and corpora differ sharply across languages. The authors explore the approach rather than claim a finished system.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-00214
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Corpus-Grounded Agentic LLMs for Multilingual Grammatical Analysis Abstract Empirical grammar research has become increasingly data-driven, but the systematic analysis of annotated corpora still requires substantial methodological and technical effort. We explore how agentic large language models (LLMs) can streamline this process by reasoning over annotated corpora and producing interpretable, data-grounded answers to linguistic questions. We introduce an agentic framework for corpus-grounded grammatical analysis that integrates concepts such as natural-language task interpretation, code generation, and data-driven reasoning. As a proof of concept, we apply it to Universal Dependencies (UD) corpora, testing it on multilingual grammatical tasks inspired by the World Atlas of Language Structures (WALS). The evaluation spans 13 word-order features and over 170 languages, assessing system performance across three complementary dimensions – dominant-order accuracy, order-coverage completeness, and distributional fidelity – which reflect how well the system generalizes, identifies, and quantifies word-order variations. The results demonstrate the feasibility of combining LLM reasoning with structured linguistic data, offering a first step toward interpretable, scalable automation of corpus-based grammatical inquiry. Keywords: corpus linguistics, grammatical analysis, large language models, agentic systems, Universal Dependencies, word-order variation Towards Corpus-Grounded Agentic LLMs for Multilingual Grammatical Analysis Matej Klemen 1 , Tjaša Arčon 1 , Luka Terčon 2 , Marko Robnik-Šikonja 1 , Kaja Dobrovoljc 1,2,3 1 University of Ljubljana, Faculty of Computer and Information Science, Večna pot 113, 1000 Ljubljana, Slovenia 2 University of Ljubljana, Faculty of Arts, Aškerčeva cesta 2, 1000 Ljubljana, Slovenia 3 Jožef Stefan Institute, Jamova cesta 39, 1000 Ljubljana, Slovenia {matej.klemen, tjasa.arcon, marko.robnik}@fri.uni-lj.si, {luka.tercon, kaja.dobrovoljc}@ff.uni-lj.si Abstract content 1. Introduction Figure 1: A simplified representation of the contrast between a traditional approach to corpus-based linguistics research (top), and our abstract solution proposal (Grammatical analysis agent, bottom). In the past four decades, linguistics has undergone a fundamental transformation from intuition-based inquiry to data-driven analysis, enabled by large-scale annotated corpora and computational tools that allow grammatical patterns to be studied systematically across languages and communicative contexts Leech 1992 ; McEnery and Hardie 2011 ; Sinclair 1991 . Yet, despite these advances, corpus-based linguistic analysis remains labor-intensive and hypothesis-driven, requiring complex scripting, specialized tools, readily accessible corpora, and methodological decisions that constrain exploratory research Gries 2009 ; Biber 2009 – limiting the scope of feasible investigations Kaunisto and Schilk 2024 . Recent advances in large language models (LLMs) with advanced reasoning capabilities Srivastava et al. 2023 ; Wei et al. 2022 open new avenues for addressing these limitations, as they have been shown to capture a broad range of grammatical regularities, perform complex linguistic reasoning, and generate structured, interpretable outputs Beguš et al. 2025 ; Jumelet et al. 2025 ; Xia et al. 2024 . Their ability to understand natural-language instructions and interface with external data sources ( Lewis et al. 2020 ) suggests that they can now serve as intelligent mediators between linguistic questions and empirical evidence, bringing a new level of automation and accessibility to corpus-based grammatical analysis. To illustrate the concept, we use the problem of determining the typical order of subject, verb, and object in a language throughout the paper. In Figure 1 , we show the contrast between the steps a user might take when trying to tackle the problem traditionally versus using an LLM-based grammatical analysis agent. In the traditional approach, the user needs to plan the experiment, select a relevant resource (e.g., a parsed corpus), select an appropriate tool (e.g., Python) and learn to use it, then analyze the resource. When using a grammatical analysis agent, the steps are abstracted away from the user, but may be inspected to validate the correctness of the process. To explore this potential, we describe the evaluation framework to mimic the motivational use-case, and propose a grammatical agent prototype to test the feasibility of the idea. Specifically, we apply it to word-order analysis in the Universal Dependencies (UD) corpora, which provide cross-linguistically consistent grammatical annotation and have become a cornerstone for large-scale, data-driven studies of grammatical structure and variation de Marneffe et al. 2021 ; Nivre et al. 2020 . More concretely, our contributions are: 1. An exploratory evaluation framework for corpus-grounded grammar analysis. We define a structured setup for evaluating grammatical analysis agents on word-order phenomena in UD data. The current demonstration framework covers 13 tasks across up to 179 languages, each with defined expected outputs. To enable automatic evaluation, we use classification accuracy, F_{1} score, and Hellinger distance to measure different aspects of distributional grammatical analysis (dominant pattern prediction, coverage of relevant patterns, and pattern distribution prediction). 2. Baseline evaluation. Using our framework, we test a strong baseline model (GPT-5) that approaches the tasks without explicit resource-grounding, instead relying on its pre-existing linguistic knowledge. We show that the model contains a varying amount of linguistic knowledge depending on the problem, but in general leaves significant room for improvement. 3. Grammatical analysis agent prototype. To verify the potential of resource grounding, we develop a prototype grammatical analysis agent (UDagent) with a simple agentic architecture: a fixed plan, a GPT-5-based code generation task-solving module, and answer construction based on the UD data. We show that despite being a conceptually simple prototype, the system already shows a performance improvement on most tasks. The remainder of this paper is organized as follows. Section 2 reviews related work. Section 3 introduces the task and evaluation framework including the design of the evaluation dataset. Section 4 describes the architecture of our agentic LLM system. Section 5 presents evaluation results across models, features, and languages. Section 6 concludes with future directions. 2. Related Work Corpus-based approaches have transformed grammatical research by enabling systematic analyses of authentic language data to uncover patterns, variation, and change Biber et al. 2021 . The growing availability of grammatically annotated corpora, such as those developed within the Universal Dependencies framework de Marneffe et al. 2021 , has been instrumental in advancing research across linguistic typology Levshina 2019 ; Yan and Liu 2023 ; Gerdes et al. 2021 , data-driven grammar induction Herrera et al. 2024 , cross-linguistic comparison of structural patterns Futrell et al. 2015 , and the study of syntactic complexity ( Berdicevskis et al. 2018 ) . These developments have established a range of computational methods that require specialized technical and statistical expertise to extract and analyze grammatical patterns from parsed corpora. More recently, advances in large language models have opened new possibilities for interacting with linguistic corpora through natural language. Initial applications demonstrate this potential in different contexts: the integration of LLMs into English-Corpora.org Davies 2025 enables users to send corpus outputs such as frequency lists or concordance lines to an LLM for summarization, explanation, and interpretation; CorpusChat supports interactive language learning through corpus-informed dialogue grounded in authentic usage examples Cheung and Crosthwaite 2025 , highlighting the complementarity between corpus-based and generative AI approaches Crosthwaite and Baisa 2023 ; and text-to-CQL systems translate natural-language requests into formal corpus queries Lu et al. 2024 . Beyond querying, LLMs have also shown promise in explaining linguistic patterns Singh et al. 2023 and inducing grammatical rules from corpus data Spencer and Kongborrirak 2025 . However, most of these applications remain task-specific and limited in scope, lacking a general, systematically evaluated framework that combines natural-language prompting with corpus-grounded reasoning to support large-scale grammatical analysis.