CANDLE: Character-level Arabic Noise Deduplication using Lightweight Encoder

Faris Alasmary, Taif Nono, Orjuwan Zaafarani, Kholood Al Tabash, Ahmad Ghannam Anas Salamah Shouq Sadah Lahouari Ghouti · 2026

Read CANDLE: Character-level Arabic Noise Deduplication using Lightweight Encoder by Faris Alasmary, Taif Nono, Orjuwan Zaafarani, Kholood Al Tabash, Ahmad Ghannam Anas Salamah Shouq Sadah Lahouari Ghouti (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24758.txt.

Introduction

Faris Alasmary and colleagues present CANDLE, a system that cleans repeated characters from Arabic text. Repetition is ambiguous in Arabic writing. Extra letters can be correct spelling, or casual elongation common in social media posts. CANDLE avoids the dictionaries and analysers that rule based fixes need. It reframes normalisation as a sequence alignment problem. A character based encoder is trained with Connectionist Temporal Classification, a method new to this task.

Three benchmarks test the system: clean newspaper text, curated ambiguous cases, and real social media text. Sentence error rate falls as low as 5.37 percent. The model beats a classification based baseline by a wide margin throughout. For speed, the six layer model is distilled into a two layer student with little loss. Normalised text also helps downstream language models. Tokeniser fertility drops by up to 12.8 percent across several Arabic tokenisers, directly lowering inference costs. Code and models are released publicly.

Catalog record

CREATORFaris Alasmary, Taif Nono, Orjuwan Zaafarani, Kholood Al Tabash, Ahmad Ghannam Anas Salamah Shouq Sadah Lahouari Ghouti
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.24758

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CANDLE: Character-level Arabic Noise Deduplication using Lightweight Encoder Faris Alasmary, Taif Nono, Orjuwan Zaafarani, Kholood Al Tabash, Ahmad Ghannam Anas Salamah Shouq Sadah Lahouari Ghouti Abstract Handling repeated characters in text can be tricky, since they can represent either the correct spelling of a word or informal character elongation often seen in social media posts. We present CANDLE, a lightweight system for character-level Arabic noise deduplication that addresses this challenge without relying on handcrafted rules, dictionaries, or morphological analyzers. At the heart of CANDLE is a novel application of Connectionist Temporal Classification (CTC) to this task, a formulation not previously explored for character deduplication, which frames normalization as a sequence alignment problem over a character-based encoder. Evaluated on three benchmarks spanning clean newspaper, manually curated ambiguous cases, and real-world social media text, the CTC model achieves a Sentence Error Rate (SER) as low as 5.37 % 5.37\% and consistently outperforms a classification-based baseline by a large margin. To reduce inference overhead, we distill the 6-layer CTC model into a 2-layer student, achieving a 3 × 3\times depth reduction with minimal performance degradation. Beyond deduplication accuracy, normalization yields a practical downstream benefit: a relative reduction in tokenizer fertility of up to 12.8 % 12.8\% across a diverse set of Arabic LLM tokenizers, directly lowering inference costs and improving context window utilization. We release all code and models publicly to support reproducibility and advance future research 1 1 1 https://github.com/abjadai/candle . \setcode utf8 1 Introduction Text normalization is an important preprocessing step in Natural Language Processing (NLP), where the goal is to reduce variation in written text without changing its meaning. One such normalization step is the removal of invalid consecutive character repetitions, that is, repeated characters that do not contribute to the meaning of the word. This issue is common across different languages, particularly in informal text on social media and online platforms, where writers frequently repeat characters within a word to express emphasis or emotion. For example, an English speaker might write "sooooo good" instead of "so good" , or an Arabic speaker might write \< مرررحبا> instead of the correct form \< مرحبا> . These repetitions have no grammatical or semantic value and can therefore be safely reduced to their standard single-character form. However, a key challenge is telling apart this kind of noisy repetition from character sequences that are linguistically correct. In Arabic, some repeated character patterns are not informal elongations but are actually valid parts of the word, and changing them would change the meaning. A good example of this is the difference between \< المملكة> ( al-mamlaka , "the kingdom") and \< الملكة> ( al-malika , "the queen"), where the extra \< م> in \< المملكة> is not noise but an essential part of the word. Removing it would silently change the meaning of the text, as shown in the following examples: 1. \< المملكة الأردنية الهاشمية> Translation: The Hashemite Kingdom of Jordan. 2. \< الملكة الأردنية رانيا> Translation: Queen Rania of Jordan. In the first example, \< المملكة> correctly means " kingdom ", while in the second, \< الملكة> means " queen ", a reading that is further supported by the proper name \< رانيا> ( Rania ). Collapsing the repeated \< م> in the first example without considering the context would produce the incorrect phrase " The Hashemite Queen of Jordan ". A good normalization system must therefore be able to tell the difference between the two cases, removing noisy repetitions while keeping the meaningful ones. In terms of how often valid repetitions occur, they rarely exceed two consecutive characters in Arabic, and three consecutive repetitions are very uncommon. The theoretical maximum, however, is five, as shown by the classical Arabic example \< ما رأينا كُكَكًاَ كَكُكَكِكُم> (literally \< ما رأينا مراكبَ كباراً مثل مراكبكم>) 1 , which means " We have never seen ships as large as yours ". Any repetition beyond that can safely be treated as noise. This paper presents a model for normalizing character-level repetitions in Arabic text. The model detects and collapses redundant consecutive character repetitions while keeping sequences that are linguistically valid. Since valid repetitions in Arabic rarely exceed two consecutive characters, the model uses a maximum threshold of two, treating anything beyond that as noise. The model works directly on character strings, without the need for phonological or morphological analysis, making it simple and efficient. To formalize the notation used throughout this paper, we define two core reduction operations. The first, reduce\_to\_1() , collapses any sequence of identical consecutive characters down to a single instance. For example, " \< الله>" becomes " \< اله>" (reducing " \< لل>" to " \< ل>"), and " good " becomes " god ". The second, reduce\_to\_2() , first applies reduce\_to\_1() and then duplicates each character so that every character appears exactly twice. For example, " \< الله>" becomes " \< االلهه>", and " good " becomes " ggoodd ". Notice that " \< ل>" and " o " do not get duplicated. We refer to the output of reduce\_to\_1() applied to a word as its reduced form . 2 Related Work Repeated-letter normalization is a key component of noisy text processing, particularly in social media, where character elongation is commonly used for emphasis or informal expression. Existing approaches can be broadly grouped into four categories: rule-based reduction methods, spell-checking and lexical similarity methods, corpus- and context-based methods, and neural sequence-to-sequence (seq2seq) models 23 . This progression from rule-based to data-driven methods reflects a continuous effort to balance efficiency, robustness, and reliance on linguistic resources. However, significant challenges remain, especially when handling highly irregular or unseen repetition patterns. 2.1 Rule-Based Reduction Approaches These methods normalize repeated characters by truncating any sequence of repeated letters to a fixed number of occurrences, typically one, two, or three. For example, 20 limits characters to a maximum of three repetitions, while 14 reduces repeated letters to a single occurrence ( AABBBCC:ABC ) and repeated syllabic patterns to two occurrences ( ABABABA:ABAB ) using handcrafted normalization rules. 3 reported that many social media normalization systems adopt similar methods. Although simple and efficient, these methods do not account for lexical validity or contextual information, and may either incorrectly alter valid words with legitimate repetitions or fail to fully remove noise from heavily elongated forms. This motivates the need for methods that can verify whether a candidate form is a valid word. 2.2 Spell-Checking and Lexical Similarity Approaches These methods extend fixed-length reduction by generating candidate word forms after removing repetitions, then selecting the most likely candidate using spell-checking or lexical similarity measures. For example, 22 applies regular expression-based reduction followed by spell-checking, while 17 uses position-aware truncation with SymSpell validation. 4 combines rule-based normalization with lexical similarity derived from word distribution, and 7 shortens words so that they do not exceed two repetitions before the lexical matching with Levenshtein distance to find the nearest match. 12 normalize Arabic social media text by removing repetitions and matching the result against a predefined dictionary of non-standard forms. While more robust than simple truncation, these methods still depend on linguistic resources and may struggle when multiple candidates have similar similarity scores, pointing to the need for broader contextual evidence to

Read the complete text