Beyond the Pipeline: Analyzing Key Factors in End-to-End Deep Learning for Historical Writer Identification

Hanif Rasyidi 1, Moshiur Farazi 3 1 · 2025

Read Beyond the Pipeline: Analyzing Key Factors in End-to-End Deep Learning for Historical Writer Identification by Hanif Rasyidi 1, Moshiur Farazi 3 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18671.txt.

Introduction

Hanif Rasyidi and Moshiur Farazi investigate why deep learning systems for identifying historical writers work less well in realistic settings than in the lab. Naming a writer from handwriting is hard: styles vary widely, documents decay, and few labeled samples exist per writer. Traditional methods use handcrafted image processing, while end-to-end systems promise to learn features directly from page images.

The authors varied preprocessing, network backbones, and post-processing, including text segmentation, patch sampling, and feature merging. Most combinations generalized poorly, especially when test writers never appeared in training, and the authors blame weak capture of fine visual detail, inconsistent patch features, and sensitivity to content noise. One simpler setup, however, matched the top-performing system. Their analysis maps the main obstacles facing robust end-to-end writer identification.

Catalog record

CREATORHanif Rasyidi 1, Moshiur Farazi 3 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-18671

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond the Pipeline: Analyzing Key Factors in End-to-End Deep Learning for Historical Writer Identification Hanif Rasyidi 1, Moshiur Farazi 3 1 Abstract This paper investigates various factors that influence the performance of end-to-end deep learning approaches for historical writer identification (HWI), a task that remains challenging due to the diversity of handwriting styles, document degradation, and the limited number of labelled samples per writer. These conditions often make accurate recognition difficult, even for human experts. Traditional HWI methods typically rely on handcrafted image processing and clustering techniques, which tend to perform well on small and carefully curated datasets. In contrast, end-to-end pipelines aim to automate the process by learning features directly from document images. However, our experiments show that many of these models struggle to generalise in more realistic, document-level settings, especially under zero-shot scenarios where writers in the test set are not present in the training data. We explore different combinations of pre-processing methods, backbone architectures, and post-processing strategies, including text segmentation, patch sampling, and feature aggregation. The results suggest that most configurations perform poorly due to weak capture of low-level visual features, inconsistent patch representations, and high sensitivity to content noise. Still, we identify one end-to-end setup that achieves results comparable to the top-performing system, despite using a simpler design. These findings point to key challenges in building robust end-to-end systems and offer insight into design choices that improve performance in historical document writer identification. I Introduction Writer identification (WI) is the process of analysing handwritten manuscripts to capture stylistic patterns unique to individual authors. Since handwriting is a form of behavioural biometrics, no two individuals produce identical writing styles, which can help with document classification [ 1 ] . WI has been applied in diverse domains such as authorship attribution, signature forgery detection, and medical diagnostics, where handwriting analysis can help identify neurodegenerative diseases by tracking changes in a patient’s writing over time. In the context of historical document analysis, WI plays a critical role in tracing document origin and enabling large-scale classification or clustering of unlabeled manuscripts. Among these applications, historical writer identification (HWI) presents particularly difficult challenges due to the age of documents, the variability in handwriting, and the physical degradation of source materials. These factors make HWI one of the most complex and error-prone scenarios for both manual and automated analysis. Even for trained paleographers and forensic handwriting analysts, the task can be unreliable. External factors such as document damage often obscure important visual cues, while handwriting inconsistency may result from changes in ink, writing tools, environmental conditions, or script conventions [ 2 ] . Experts typically rely on subtle visual characteristics such as stroke curvature, spacing, and letter formation, but their assessments can be subjective and inconsistent, especially when dealing with incomplete or degraded documents. Efforts to automate HWI usually begin with pre-processing steps such as binarization and segmentation to isolate areas of interest (AOI). These are followed by handcrafted feature extraction techniques, including SIFT, HOG, oBIFs, or GLCM, and by measuring similarity through distance metrics or clustering algorithms. Although these methods have shown strong performance on well-curated datasets, they tend to be brittle and sensitive to noise. Their reliance on meticulous tuning and dataset-specific configurations often limits their scalability and generalisation, particularly in zero-shot scenarios involving unseen writers. End-to-end deep learning offers a promising alternative by learning features directly from document images. With the rise of powerful convolutional and transformer-based backbones, several recent studies have applied such models to HWI tasks. Yet, their success is often limited to controlled scenarios such as word-level WI or datasets with similar text content between train and test splits. For example, some models rely on handcrafted AOI selection [ 3 ] or require tightly constrained word-level annotations [ 4 ] , which do not generalise to document-level HWI. Other approaches enhance retrieval accuracy through re-ranking or pre-training on related domains [ 5 , 6 , 7 ] , but these methods are hard to reproduce and often entangle deep learning with domain-specific heuristics. In this paper, we analyse the key factors that influence the performance of end-to-end deep learning pipelines for historical writer identification (HWI). Our contributions are as follows: • We explore multiple configurations of end-to-end deep learning pipelines to identify setups that offer a good balance between simplicity and performance. • We evaluate the impact of various pre-processing strategies, including SIFT-based keypoint detection, document binarisation, and text region selection, on challenging historical document datasets. • We investigate how different backbone architectures, loss functions, and post-processing techniques affect overall performance, and compare them with state-of-the-art, more complex approaches on widely used benchmark datasets. Rather than aiming to find a single best-performing model, our goal is to understand which design choices contribute most to performance variation, particularly in zero-shot settings involving unseen writers and diverse document layouts. Our findings indicate that unstable feature representations, inconsistent patch-level outputs, and weak generalisation are key obstacles to building effective end-to-end HWI systems. These insights highlight the importance of integrating deep learning models with domain-specific pre- and post-processing steps to improve robustness in real-world scenarios. II Related Works Several historical writer identification methods have been published over the years, proposing various image-processing techniques and using learning models. The process usually starts with pre-processing, in the form of binarization, noise removal, segmentation (letter, word, lines or paragraph level), and feature AOI selection (keypoints or specific contour in the image). The result of pre-processing then goes into feature extraction using image processing or a learning model, followed by post-processing to find similarity between documents. II-A Image Processing Approach To capture writing style with non-standard patterns, various image processing techniques are used to transform the RGB/grayscale image into different vectors representing the meaningful abstraction on either a local or global scale, improving the aggregation of information that previously was hardly available from standard human observation. The local feature extraction focuses on capturing information containing the handwriting marker such as pen strokes, shape variations, and unique key points, using techniques such as HOG (Histogram of Oriented Gradients) [ 8 , 9 ] , oBIFs (Oriented Basic Image Features) [ 10 , 11 ] , or SIFT (Scale-Invariant Feature Transform) [ 12 , 13 , 14 ] . On the other hand, global feature extraction, like wavelet and GLCM (Grey-Level Co-Occurrence Matrix), captures global information in the form of large patterns and spatial relationships between lines on the entire image. With HWI tends to focus on the subtle writing details of each author, the local feature extraction approach is generally preferable, paired with image pre-processing and post-processing, like clustering, to reduce the amount of information due to other noise from the historical document. II-B Deep Learning Approach The deep learning method generally uses different types of feature extraction backbone [ 15 , 16 ] to process information directly from the historical image. This approach gives more freedom compared to the image-processing approach since the model is not bound to the pre-determined features when finding the area of interest in the writing and learning the hidden writer characteristics from each author. Although this approach managed to get the best result in many classification tasks, using one for HWI is not easy due to the scarcity of

Read the complete text