Bridging the Gap: Fusing CNNs and Transformers to Decode the Elegance of Handwritten Arabic Script

Chaouki BOUFENAR, Mehdi Ayoub RABIAI, Boualem Nadjib ZAHAF, Khelil Rafik OUARAS · 2025

Read Bridging the Gap: Fusing CNNs and Transformers to Decode the Elegance of Handwritten Arabic Script by Chaouki BOUFENAR, Mehdi Ayoub RABIAI, Boualem Nadjib ZAHAF, Khelil Rafik OUARAS (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15023.txt.

Introduction

Chaouki Boufenar and three co-authors blend two AI styles to read handwritten Arabic in a 2025 paper. Arabic script shifts shape by where a letter sits in a word, which makes machine reading hard. Their blend pairs a convolutional network, strong on local strokes, with a transformer, strong on context across the whole word.

They tested custom and tuned models, then fused the best two by trust scores. The fused model read letters at over ninety six percent accuracy on a known data set. Place in word calls ran even higher. The authors say the two styles cover each other's blind spots. They offer the blend as a base for real world Arabic text tools.

Catalog record

CREATORChaouki BOUFENAR, Mehdi Ayoub RABIAI, Boualem Nadjib ZAHAF, Khelil Rafik OUARAS
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-15023

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bridging the Gap: Fusing CNNs and Transformers to Decode the Elegance of Handwritten Arabic Script Chaouki BOUFENAR, Mehdi Ayoub RABIAI, Boualem Nadjib ZAHAF, Khelil Rafik OUARAS Abstract Handwritten Arabic script recognition is a challenging task due to the script’s dynamic letter forms and contextual variations. This paper proposes a hybrid approach combining convolutional neural networks (CNNs) and Transformer-based architectures to address these complexities. We evaluated custom and fine-tuned models, including EfficientNet-B7 and Vision Transformer (ViT-B16), and introduced an ensemble model that leverages confidence-based fusion to integrate their strengths. Our ensemble achieves remarkable performance on the IFN/ENIT dataset, with 96.38% accuracy for letter classification and 97.22% for positional classification. The results highlight the complementary nature of CNNs and Transformers, demonstrating their combined potential for robust Arabic handwriting recognition. This work advances OCR systems, offering a scalable solution for real-world applications. I Introduction The recognition of handwritten Arabic script presents a formidable challenge in the field of optical character recognition (OCR). Unlike Latin scripts, Arabic handwriting is characterized by dynamic letter forms, intricate connections, and contextual variations, significantly complicating accurate recognition. Traditional OCR systems, designed primarily for printed text, often falter when confronted with the fluidity and variability inherent in handwritten Arabic. This limitation not only hinders the digitization of historical manuscripts and personal documents but also restricts accessibility for individuals and institutions relying on handwritten records. In this study, we address these challenges by proposing a novel, intelligent system that leverages the complementary strengths of convolutional neural networks (CNNs) and Transformer-based architectures. CNNs, renowned for their ability to capture fine-grained spatial features, are integrated with Transformers, which excel in modeling global dependencies and contextual relationships. This hybrid approach aims to mirror human perception, enabling the system to analyze both the structural and semantic aspects of Arabic handwriting. By pushing the boundaries of computational intelligence, our goal is to achieve unprecedented accuracy and adaptability in real-world scenarios, thereby bridging the gap between human and machine understanding of handwritten Arabic script. To ensure robust performance, we utilize a customized dataset based on the IFN/ENIT dataset [ 1 ] , a benchmark for Arabic handwriting recognition, which includes a diverse range of handwriting styles and significant intra- and inter-writer variability. Additionally, we employ a comprehensive data augmentation pipeline, incorporating techniques such as elastic deformation, geometric transformations, and Gaussian noise, to enhance model generalization and robustness to real-world variations. These preprocessing steps ensure that our models are well-equipped to handle the complexities of Arabic script, from subtle stroke variations to broader contextual dependencies. II Related Work Handwritten Arabic script recognition presents unique challenges due to its cursive nature, contextual variations, and diacritics. Traditional OCR methods relied on handcrafted features and statistical models like Hidden Markov Models (HMMs) and hybrid HMM/ANN systems [ 2 , 3 ] , but struggled with the high variability of handwriting. The rise of deep learning introduced convolutional neural networks (CNNs) as powerful feature extractors, achieving high accuracy on datasets like AHCD and IFN/ENIT [ 4 ] . However, CNNs face limitations in capturing long-range dependencies, which are crucial for cursive scripts. To address this, Transformer-based models, such as HATFormer [ 5 ] , leverage self-attention mechanisms for improved recognition. Hybrid approaches combining CNNs with recurrent neural networks (RNNs) [ 6 ] further enhanced performance by integrating local and global features. Recent research has introduced various deep learning architectures to tackle these challenges, each bringing unique innovations to Arabic handwriting recognition. From Transformer-based models to hybrid CNN-RNN frameworks and novel ensemble methods, different approaches have been proposed to enhance accuracy and robustness. The following works illustrate key advancements in this field. II-A HATFormer: Historic Handwritten Arabic Text Recognition with Transformers HATFormer [ 5 ] propose a Transformer-based encoder-decoder architecture tailored for Arabic handwritten text recognition. Leveraging the Transformer’s attention mechanism, HATFormer captures spatial contextual information to address challenges inherent in Arabic script, such as cursive characters and diacritics. Customizations include an image processor for effective ViT preprocessing, a text tokenizer for compact Arabic text representation, and a training pipeline accommodating limited historical Arabic handwriting data. HATFormer achieved a Character Error Rate (CER) of 8.6% on the largest public historical handwritten Arabic dataset, marking a 51% improvement over existing baselines. Additionally, it attained a CER of 4.2% on a large private non-historical dataset. II-B Arabic Handwriting Recognition System Using Convolutional Neural Network This study explores various datasets for Arabic handwriting recognition, including IFN/ENIT and AHDB. The authors review multiple deep learning approaches, emphasizing the effectiveness of CNNs in recognizing Arabic numerals and characters. Notably, a CNN model achieved 94.9% accuracy on the AHCD database, which contains 16,800 handwritten Arabic characters [ 7 ] . The CNN model demonstrated high accuracy in recognizing Arabic characters, highlighting the potential of deep learning techniques in this domain. II-C An End-to-End OCR Framework for Robust Arabic Handwriting Recognition Using a Novel Transformers-Based Model Aly Mostafa and al. [ 8 ] present an end-to-end Optical Character Recognition (OCR) framework employing Vision Transformers (ViT) as an encoder and a vanilla Transformer as a decoder, eliminating the need for CNNs in feature extraction. The study also introduces a 270 million-word multi-font corpus of Classical Arabic with diacritics to enhance training. The proposed model achieved a CER of 4.46%, demonstrating the efficacy of Transformer-based architectures in Arabic handwriting recognition. II-D Hybrid Neutrosophic Deep Learning Model for Enhanced Arabic Handwriting Recognition This paper introduces a hybrid approach combining Neutrosophic Sets with deep learning models, specifically CNNs integrated with Bidirectional Recurrent Neural Networks (Bi-LSTM and Bi-GRU). This integration aims to enhance the recognition of handwritten Arabic characters by capturing both spatial features and temporal dynamics [ 9 ] . The NS_CNN_Bi-LSTM model achieved an accuracy of 92.38% on the Hijjaa dataset, while the NS_CNN_Bi-GRU model attained 97.38% accuracy on the AHCD dataset, outperforming previous deep learning approaches. II-E Advancing Arabic Handwriting Recognition with Convolutional and Recurrent Neural Network Ensembles The study investigates the application of various deep learning techniques, including CNNs, LSTMs, Bi-LSTMs, GRUs, and Bi-GRUs, to Arabic handwriting recognition. The models were tested on the AHCD and Hijjaa datasets, and their performances were compared using metrics such as Precision, Recall, F1-Score, and Accuracy [ 6 ] . The Bi-GRU model achieved the highest performance on the AHCD dataset with an accuracy rate of 95.7%, while the CNN model achieved an accuracy rate of 86.3% on the Hijjaa dataset. III Overview of the IFN/ENIT Dataset The IFN/ENIT dataset [ 1 ] is a widely recognized benchmark for handwritten Arabic word recognition, extensively used in Optical Character Recognition (OCR) research. It comprises 32,489 handwritten Arabic word images stored in BMP format, primarily representing Tunisian city names. III-A Dataset Structure The dataset is divided into five subsets ( set_a, set_b, set_c, set_d, and set_e ), each containing unique word samples written by different individuals. This ensures a diverse range of handwriting styles. III-B Variability in Handwriting The IFN/ENIT presents significant intra-writer and inter-writer variability, including differences in stroke thickness, letter shapes, spacing, and writing orientation. This variability makes it a challenging benchmark for OCR models, particularly in fine-grained classification tasks such as letter segmentation and position recognition. Fig. 1: Example of a word image from the IFN/ENIT dataset. IV Preprocessing Pipeline The preprocessing pipeline

Read the complete text