TrOCR for Medieval HTR: A Systematic Ablation Study with Cross-Dataset Validation

Sachin Sharma, Michele Flammini, Federico Simonetta · 2026

Read TrOCR for Medieval HTR: A Systematic Ablation Study with Cross-Dataset Validation by Sachin Sharma, Michele Flammini, Federico Simonetta (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24302.txt.

Introduction

Sharma, Flammini, and Simonetta fine-tune a modern handwriting recogniser on medieval manuscripts, and carefully ablate what helps. TrOCR, a transformer-based recognition model, is pretrained on modern text. Thirteenth-century Italian script is thus a foreign visual domain, and historical datasets are small. The team varies three controllable choices: contrast normalisation, data augmentation, and freezing layers during training. The testbed is a manuscript known as the Cortonese.

The best setup reaches a character error rate near eight percent on the Cortonese. Across thirteen setups, freezing up to three encoder layers or six decoder layers costs nothing significant. Deeper freezing steadily harms accuracy. Removing it barely hurts when training is strong. Good training, it seems, can replace image preprocessing. Running the grid on the public READ-16 benchmark shows decoder freezing thresholds transfer between datasets better than encoder ones. Sharma's 2026 ablation gives digital humanists an evidence-based recipe.

Catalog record

CREATORSachin Sharma, Michele Flammini, Federico Simonetta
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.24302

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TrOCR for Medieval HTR: A Systematic Ablation Study with Cross-Dataset Validation Sachin Sharma, Michele Flammini, Federico Simonetta Abstract Fine-tuning transformer-based handwritten text recognition (HTR) models on medieval manuscripts is challenging because these models are pre-trained on modern text and must adapt to a very different visual domain. This paper studies how three controllable fine-tuning choices (contrast normalization, data augmentation, and layer freezing) affect recognition accuracy when adapting TrOCR to small historical datasets. We run controlled experiments on a 13th-century Italian manuscript (I-CT 91 “Cortonese”) and replicate the same experimental grid on the public READ-16 benchmark as robustness evidence. On Cortonese, our best configuration achieves 8.03% character error rate (CER). Statistical comparisons across 13 configurations show that freezing up to three encoder layers or six decoder layers does not significantly harm accuracy, while deeper freezing becomes progressively detrimental. Removing contrast normalization (CLAHE) yields 7.84% CER, comparable to a domain-specialized baseline, suggesting strong optimization can reduce reliance on image preprocessing. Cross-dataset validation on READ-16 shows that decoder freezing thresholds transfer more robustly than encoder thresholds, and combined freezing strategies require dataset-specific re-validation. Finally, we use Grad-CAM gradient attributions and decoder cross-attention maps to diagnose error patterns and failure modes revealed by the ablations. Source code is available at https://github.com/LaudareProject/TrOCR-analysis 1 Introduction Handwritten text recognition (HTR) is a prerequisite for turning medieval manuscripts into searchable, analyzable corpora, yet performance remains fragile. Relative to modern printed documents, manuscripts combine shifting letterforms, pervasive abbreviations, degraded supports (faded ink, bleed-through, staining), and irregular layouts [ 22 ] . Systems tuned to clean print therefore generalize poorly in this domain [ 16 , 15 ] . Recent transformer-based HTR models have improved modern handwriting recognition. TrOCR [ 14 ] , for instance, performs strongly in contemporary domains, but medieval material lies far from its pre-training distribution in handwriting style, orthographic conventions, and linguistic structure [ 2 , 1 ] . We select TrOCR as the baseline because it is a strong and widely used encoder–decoder HTR model with public checkpoints, making its fine-tuning behavior practically relevant. At the same time, historical collections often provide only limited supervision (typically a few thousand labeled lines), so the details of fine-tuning can dominate the final outcome [ 24 , 17 ] . The historical HTR literature includes a wide range of application-focused studies [ 25 , 6 , 26 ] , and recent work confirms that transformer-based models can be adapted to historical documents [ 23 , 27 , 7 , 11 ] . However, reported gains are often hard to interpret because preprocessing, augmentation, and optimization are frequently adjusted together; controlled ablations remain relatively uncommon [ 17 ] . It is also unclear how far a single hyperparameter configuration transfers across scripts, languages, and periods. Interpretability is similarly underexplored. Attention visualizations are a common starting point; Donut, for example, reports decoder cross-attention maps with emergent localization behavior [ 10 ] . Yet attention weights are not explanations by default [ 3 ] . Gradient-based attributions such as Grad-CAM [ 20 ] are more tightly coupled to the training objective, but attention- and gradient-derived signals are still rarely used as quantitative diagnostics in OCR/HTR [ 10 , 12 ] . Within Document Analysis Systems, this work addresses the line-level HTR component of a broader transcription pipeline: manuscript image \rightarrow segmentation \rightarrow line-level HTR \rightarrow correction/review \rightarrow document-level transcription use. We study this question with controlled ablations that isolate preprocessing, data augmentation, and encoder/decoder layer freezing. Overall, this work contributes practical recommendations for TrOCR users: 1. We provide controlled ablations on a 13th-century Italian manuscript, isolating preprocessing, augmentation, and encoder–decoder freezing effects on CER and generalization, with accompanying statistical significance tests. 2. We provide a full 13-configuration ablation replicated on READ-16 with line-level statistical validation, quantifying which findings transfer across scripts, languages, and centuries and which require dataset-specific re-validation. 1 1 1 The source code used for the experiments is available at https://github.com/LaudareProject/TrOCR-analysis . The dataset is currently under review and will be released upon completion of the review. 3. a qualitative diagnostic analysis using Grad-CAM and cross-attention to interpret ablation-linked error patterns and failure modes. In order to test whether fine-tuning conclusions are specific to a single corpus or transfer across typologically distant collections, we perform a cross-dataset validation using two datasets that differ along several complementary axes: script, language, time period, and document type (I-CT 91 and READ-16). We do not claim to outperform all existing HTR systems. Instead, we provide a controlled ablation study and actionable fine-tuning guidance for practitioners working with small medieval HTR datasets. 2 Dataset and Experimental Setup 2.1 Dataset Figure 1: Example of a page from the I-CT 91 “Cortonese” manuscript All experiments were performed on the I-CT 91 “Cortonese” manuscript (Fig. 1 ) [ 21 ] , an Italian laude source dating from the 13th century. The dataset comprises 259 manuscript pages, 2,517 annotated text lines, 14,214 words, and 75,667 characters. Each line is provided with verified ground truth transcriptions. The average image resolution is 2082 \times 2717 with 300 DPI and average text line size of 1352.85 \times 159.64. The manuscript presents multiple paleographic challenges, including the presence of two different scribal hands, the coexistence of textual and neumatic notation, and substantial physical degradation such as faded ink and parchment staining. These characteristics make the collection low-resource medieval test case for line-level HTR adaptation under real-world manuscript conditions. To ensure reproducibility and fair evaluation, we adopted one of the standardized splits provided by our project. The partition preserves page-level separation to prevent data leakage between training and testing subsets. The dataset includes 1,666 text lines for training ( 66.2\% ), 397 for validation ( 15.8\% ), and 454 for testing ( 18\% ). Unlike random splitting, this configuration guarantees consistent comparisons across studies and future reproducibility for subsequent research on the same material. The exact split indices and all experimental source code are publicly available in the code repository to facilitate replication once the dataset is released. 3 Methodology In order to define a rigorous protocol, we define the main set the following research question: when adapting a pre-trained TrOCR model to a small medieval HTR dataset, which architectural choices impact the CER metric the most, and to which extent the impact is measurable? To investigate this question, we examine three controllable fine-tuning levers: a) preprocessing, b) data augmentation, and c) parameter-efficient fine-tuning through layer freezing. These axes were selected because they are common, controllable design decisions in low-resource HTR pipelines and directly trade off accuracy, compute, and manual pipeline effort. All other training factors are held constant throughout the experiments. The experimental design modifies one lever at a time relative to a fixed full fine-tuning baseline, thereby enabling a clean attribution of performance differences to individual methodological choices. Preprocessing Pipeline: The preprocessing pipeline was designed to enhance image readability and improve OCR robustness under varying manuscript conditions. Contrast enhancement was performed using Contrast Limited Adaptive Histogram Equalization (CLAHE) with a clip limit of 2.0 and an 8 \times 8 tile grid [ 28 , 19 ] . For comparison, standard histogram equalization was also considered during first experimentation. Deskewing was applied using a projection profile method constrained within \pm 5^{\circ} ,

Read the complete text