Integrated ensemble of BERT- and features-based models for authorship attribution in Japanese literary works

Taisei Kanda, Mingzhe Jin, Wataru Zaitsu · 2025

Read Integrated ensemble of BERT- and features-based models for authorship attribution in Japanese literary works by Taisei Kanda, Mingzhe Jin, Wataru Zaitsu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-08527.txt.

Introduction

Taisei Kanda and co-authors join two model families to name the author of Japanese literary works. Their paper from 2025 deals with authorship attribution, a task long done with statistics on style traits. New pre trained language models do well on large short text sets, but their worth on small samples was little tested.

The team built an ensemble that blends a BERT based model with classic trait based classifiers. Two corpora were used, each with works by ten authors to sort. The language model did well even on small samples. Each ensemble beat its single models, and the full blend gained most, with a jump of about fourteen points on the corpus left out of pre training.

Catalog record

CREATORTaisei Kanda, Mingzhe Jin, Wataru Zaitsu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-08527

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Integrated ensemble of BERT- and features-based models for authorship attribution in Japanese literary works Taisei Kanda, Mingzhe Jin, Wataru Zaitsu Abstract Traditionally, authorship attribution (AA) tasks relied on statistical data analysis and classification based on stylistic features extracted from texts. In recent years, pre-trained language models (PLMs) have attracted significant attention in text classification tasks. However, although they demonstrate excellent performance on large-scale short-text datasets, their effectiveness remains under-explored for small samples, particularly in AA tasks. Additionally, a key challenge is how to effectively leverage PLMs in conjunction with traditional feature-based methods to advance AA research. In this study, we aimed to significantly improve performance using an integrated integrative ensemble of traditional feature-based and modern PLM-based methods on an AA task in a small sample. For the experiment, we used two corpora of literary works to classify 10 authors each. The results indicate that BERT is effective, even for small-sample AA tasks. Both BERT-based and classifier ensembles outperformed their respective stand-alone models, and the integrated ensemble approach further improved the scores significantly. For the corpus that was not included in the pre-training data, the integrated ensemble improved the F1 score by approximately 14 points, compared to the best-performing single model. Our methodology provides a viable solution for the efficient use of the ever-expanding array of data processing tools in the foreseeable future. Keywords: Bidirectional encoder representations from transformers (BERT), Stylometric features, Authorship attribution (AA), Text classification, Integrated ensemble. 1 Introduction Authorship attribution (AA) entails assigning authorship to texts with unknown authorship [1, 2, 3]. Various studies on AA have been conducted, tracing back more than 100 years. Mendenhall [4] counted the number of letters in each word used in a sentence, analyzed the relative frequency curves, and demonstrated that the curves varied among authors and could be a distinctive characteristic of each author. Mendenhall [5] also demonstrated that Shakespeare predominantly used four-letter words, whereas Bacon favored three-letter words. This finding challenged the theory that Bacon authored a series of satirical plays under the pseudonym "Shakespeare" to protest against the oppressive government. These features are referred to as stylistic features. They are distinctive techniques or devices that an author uses to create a particular effect in a text. These are woven throughout the work. Before the 1950s, word, sentence, and paragraph lengths, which were easy to quantify, were mainly used for statistical analysis. With the development of computational environments, many scholars have proposed extracting stylistic features from text based on aspects such as character, word, part-of-speech structure, grammar, and syntax [2, 6-9]. Stylistic features that are mechanically aggregated from texts contain a lot of noise. Stylistic features, subsequently referred to as features in this paper, are language-dependent. For example, Japanese or Chinese differ from Western languages in character forms, writing styles, lack of segmentation, and grammatical structures. One of the most evident ways in which Japanese or Chinese are unique is the character forms and the fact that they are not divided into words. Therefore, the elements that appear as features, such as characters, words, and phrases, also vary depending on the segmentation method. AA was first performed in stylistic studies of literary works. Over time, it has also been applied to detect fake news, address authorship issues, identify plagiarism, and investigate matters in criminal and civil law [9, 10, 11]. With the rapid development of computer science, powerful machine learning classifiers and pre-trained language models (PLMs) are being consecutively developed, and the AA environment is changing rapidly. Several classifiers are now commonly used for text classification, including penalized logistic regression, support vector machine (SVM), random forest (RF), boosting methods (such as AdaBoost, XGBoost), neural network approaches, and PLMs such as bidirectional encoder representations from transformers (BERT) and its derivatives, RoBERTa and DeBERTa. However, it is difficult to determine the best model. Moreover, an important issue for end users is how to adapt such tools to their specific tasks. Against this backdrop, this study focused on significantly improving AA scores in small-scale samples and literary works by employing an integrated ensemble that combines traditional feature- and classifier-based methods with PLM-based methods. The remainder of this paper is organized as follows. Section 2 presents a review of pertinent literature, identifies the limitations of previous studies, and outlines the research questions. Sections 3 and 4 present the experimental procedure and study results, respectively. Sections 5 and 6 are the discussion and conclusions, respectively. 2 Related research Multivariate data analysis methods that use features extracted manually or automatically from texts have been used in AA tasks since 1960. These methods include unsupervised techniques, such as principal component analysis, correspondence analysis, and clustering, as well as supervised techniques, such as linear and nonlinear discriminant analysis. Since the 1990s, neural networks [12, 13], SVM, RF, boosting classifiers have been employed [2, 3, 14, 15]. Jin and Murakami [14] demonstrated that RF is more effective than SVM for noisy data. They also analyzed the decrease in the RF and SVM scores with decreasing sample size. Liu and Jin [15] conducted a comparative analysis of the genre and author classifications of mixed-genre texts using 14 feature datasets (character n -gram, n=1,2,3 ; POS tag n -gram, n=1,2,3 ; token n -gram, n=1,2,3 ; token and POS tag n-gram, n=1,2,3 ; phrase pattern; comma position ) and seven classifiers (SVM, RF, AdaBoost, HDDA, LMT, XGBoost, and Lasso). The results indicate that features and classifier scores vary across different cases. In other words, even within the same corpus, variations arise where combinations such as character bigrams with the RF classifier demonstrate higher score, whereas token unigrams paired with the AdaBoost classifier may show the best performance. Regarding the use of structured data features and classifiers in AA, there have been studies on methods for feature extraction and classifier arrangements; however, there has been no significant progress in score improvement. Considering this, Jin [16] proposed an ensemble approach that employed multiple feature vectors (character bigram, token and POS tag uigram, POS tag bigram, phrase pattern) and classifiers (RF, SVM, LMT, ADD, DWD, HDD). This approach was tested on three text types: novels, student essays, and personal diaries. Four types of features and six classifiers were used to classify the texts. An ensemble of the 24 results derived from four types of features and six classifiers demonstrated excellent validity. The scores of ensembles of multiple classification models generally equal or exceed the best score achieved by any individual model; moreover, ensembles are more robust than individual models. Robustness is crucial for drawing conclusions to real-world problems. Hence, ensembles are gradually finding practical applications [17,18]. Bacciu et al. [17] performed text classification in a soft-voting ensemble using the classification results of multiple different features with a single classifier (SVM). Lasotte et al. [18] used the results of four classifiers with a single feature dataset to detect fake news and demonstrated the effectiveness of ensembles. In 2013, Mikolov et al. [19] proposed an algorithm for embedding words or sentences into vectors, demonstrating an approach for classifying text without extracting its features. In 2018, Google released a BERT model pre-trained on a large corpus of English text (Wikipedia + BookCorpus); the model achieved state-of-the-art (SOTA) performance on several natural language processing (NLP) tasks [20]. BERT

Read the complete text