Comparing LLM-generated and human-authored news text using formal syntactic theory

Olga Zamaraeva olga.zamaraeva, Dan Flickinger danflick, Francis Bond francis.bond, Carlos Gómez-Rodríguez carlos.gomez, [-1cm] Universidade da Coruña, CITIC ( @udc.es ) · 2025

Read Comparing LLM-generated and human-authored news text using formal syntactic theory by Olga Zamaraeva olga.zamaraeva, Dan Flickinger danflick, Francis Bond francis.bond, Carlos Gómez-Rodríguez carlos.gomez, [-1cm] Universidade da Coruña, CITIC ( @udc.es ) (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01407.txt.

Introduction

Olga Zamaraeva, Dan Flickinger, Francis Bond, and Carlos Gomez-Rodriguez wrote this linguistics paper, published in two thousand twenty-five, on how machine writing differs from human writing at the level of grammar. They compare New York Times-style text from six large language models with real Times prose.

The lens is a formal grammar theory, Head-driven Phrase Structure Grammar, which parses each sentence into a detailed structure. Counting which grammar types show up, and how often, exposes systematic gaps between the two kinds of text. The models' writing leans on a narrower slice of the grammar than the humans' does. The authors say such counts give a sharper picture than surface word checks. The study speaks to both language model builders and students of human style.

Catalog record

CREATOROlga Zamaraeva olga.zamaraeva, Dan Flickinger danflick, Francis Bond francis.bond, Carlos Gómez-Rodríguez carlos.gomez, [-1cm] Universidade da Coruña, CITIC ( @udc.es )
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-01407

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Comparing LLM-generated and human-authored news text using formal syntactic theory Olga Zamaraeva olga.zamaraeva, Dan Flickinger danflick, Francis Bond francis.bond, Carlos Gómez-Rodríguez carlos.gomez, [-1cm] Universidade da Coruña, CITIC ( @udc.es ) Abstract This study provides the first comprehensive comparison of New York Times-style text generated by six large language models against real, human-authored NYT writing. The comparison is based on a formal syntactic theory. We use Head-driven Phrase Structure Grammar (HPSG) to analyze the grammatical structure of the texts. We then investigate and illustrate the differences in the distributions of HPSG grammar types, revealing systematic distinctions between human and LLM-generated writing. These findings contribute to a deeper understanding of the syntactic behavior of LLMs as well as humans, within the NYT genre. \forestapplylibrarydefaults linguistics 1 Introduction Studying linguistic properties of LLM-generated text and comparing it to human-authored text is a topic of growing interest in the field of natural language processing (NLP). Previous research has predominantly focused on training classifiers (is the text LLM-generated or no); a few studies include an analysis of differences in vocabulary distribution, use of dependency structures, or sentiment properties of the text (see § 2 ). In this study, we systematically analyze grammatical differences of LLM-generated vs. human-authored text through the lens of a formal syntactic theory developed for linguistic research independently of NLP. 1 1 1 Annotation schemes such as Universal Dependencies ( Nivre et al. 2016 , UD:) or Penn Treebank ( Marcus et al. 1993 , PTB:) are related to syntactic theory but they have been developed as guidelines for hand-annotating corpora specifically for NLP. As such, they are less detailed and consistent than a formal theory and less independent from NLP tasks themselves. Using a formal theory for analysis and evaluation is a way to overcome some of the biases that arise from using tools developed directly in the context of designing NLP tasks. We hope this will lead to further systematic discoveries about grammatical properties of LLM-generated text and how they differ from human-authored text. In this paper, we use the broad-coverage English Resource Grammar ( Flickinger 2000 ; Flickinger 2011 ) to analyze texts in the New York Times genre. 2 Related work Our study is concerned with the analysis of the grammatical properties of LLM-generated texts as compared to human-authored texts. Here, we review the literature with a similar focus. This leaves out of scope papers concerned with building classifiers or with sentiment and semantic analysis. Muñoz-Ortiz et al. 2024 include a study of syntactic and vocabulary diversity in NYT-style news. They conclude that measurable differences can be detected, including at the level of grammar, and that human-authored texts exhibit more variety of vocabulary, shorter constituents, and more optimized dependency distances. Narayanan et al. 2024 use the Universal Sentence Encoder ( Cer et al. 2018 , USE:) to compare human-authored and AI-generated code explanations and find statistical differences, though without linguistic analysis. Sandler et al. 2024 base the comparison on ChatGPT-human dialogues, using primarily lexical features, not syntactic, and find greater diversity in texts written by humans. Notably, they use dictionary-style features and not just raw vocabulary. So do Alvero et al. 2024 , who compare college application essays (submitted in 2016-2017) with texts generated by GPT-3.5 and GPT-4. They find that human authors show more variety in e.g. verb usage. Juzek and Ward 2025 study the vocabulary of LLMs linking it to the increase of use in certain vocabulary items in scientific abstracts (e.g. the word ‘delve’). Park et al. 2025 perform a statistical comparison by clustering linguistic features (this is necessary to obtain statistically significant results in the context of multiple comparisons between many features). They conclude that LLM-generated texts have a distinct statistical footprint from human-authored text. Shaib et al. 2024 compare strings of POS-tags, which they call “syntactic templates”, finding that LLMs tend to repeat these templates more than humans do. Finally, several studies base the comparison on a set of linguistic features proposed for rhetoric styles by Biber 1991 ( Biber 1991 , Biber 1995 ) and Biber and Conrad 2019 . In particular, Reinhart et al. 2024 show that LLMs prefer certain grammatical constructions and thus struggle to match styles that do not employ them (according to Biber 1991 ). The constructions include participial clauses, ‘that’-subject clauses, nominalization, phrasal and clause coordination. Sardinha 2024 also uses the “Biber features”. This study is perhaps the closest in spirit to ours, since it uses an independently developed linguistic framework and presents examples of the differences found. 3 Methodology The central idea of our methodology is to apply formal syntactic theory to analyzing structural properties of texts generated by LLMs as compared to human-authored texts. We use the HPSG theory of syntax (§ 3.1 ), specifically its implementation as the English Resource Grammar (§ 3.2 ), the largest available implementation of a formal grammar in terms of its coverage over naturally occurring text (in any language and in any theory). While applying such methodology implies the investment in building the grammar, the HPSG theory and the formalism were developed precisely to be used for a wide variety of languages. The cross-linguistic applicability of the theory has been continuously tested in the context of the Grammar Matrix ( Bender et al. 2002 ; Bender et al. 2010 ; Zamaraeva et al. 2022 ) and the AGGREGATION Bender et al. 2020 ; Howell and Bender 2022 projects. 3.1 HPSG Head-driven Phrase Structure Grammar ( Pollard and Sag 1994 , HPSG:) is a formal theory of syntax that uses a fully explicit formalism, so it can be implemented on the computer in its entirety as a grammar which then maps sentences to complete structures automatically, while remaining fully consistent and interpretable. The theory represents syntactic structure and elements of the syntax-semantic interface (dependencies, quantifier scope, information structure) as a complex graph, which can also be visualized as an attribute-value matrix of features and their values (such as the feature head having a value noun ). HPSG assumes lexical types which can house multiple lexical entries, and, unlike raw vocabulary forms, lexical types contain information about syntactic properties of words. The grammar as a whole (the lexicon included) is a hierarchy of types. Figure 1 shows a very small and simplified portion of the HPSG type hierarchy, with only two features ( head and comps , complement list). This part pertains to the lexicon and lexical types. The noun ‘law’ can behave in different ways syntactically, which motivates two lexical entries belonging to two different types (which may house other nouns as well). In § 5 we report on how this word is one of the examples of differences in human-authored and LLM-generated texts that we examined. The real type hierarchy, such as the one in the ERG (§ 3.2 ), consists of hundreds of types with dozens of features, allowing us to examine grammatical properties of sentences in detail. {forest} Figure 1: Part of the HPSG type hierarchy (simplified; adapted from ERG). Table 1: Datasets: reproduced in full from Table 1 in Muñoz-Ortiz et al. 2024 , plus the information on Redwoods. Dataset # Sent. in dataset Model size Training tokens Data sources LLaMa 37,825 7B 1T English CommonCrawl

Read the complete text