Curriculum Learning and Pseudo-Labeling Improve the Generalization of Multi-Label Arabic Dialect Identification Models Ali Mekky, Mohamed El Zeftawy, Lara Hassan, Amr Keleg, Preslav Nakov Abstract Being modeled as a single-label classification task for a long time, recent work has argued that Arabic Dialect Identification (ADI) should be framed as a multi-label classification task. However, ADI remains constrained by the availability of single-label datasets, with no large-scale multi-label resources available for training. By analyzing models trained on single-label ADI data, we show that the main difficulty in repurposing such datasets for Multi-Label Arabic Dialect Identification (MLADI) lies in the selection of negative samples, as many sentences treated as negative could be acceptable in multiple dialects. To address these issues, we construct a multi-label dataset by generating automatic multi-label annotations using GPT-4o and binary dialect acceptability classifiers, with aggregation guided by the Arabic Level of Dialectness (ALDi). Afterward, we train a BERT-based multi-label classifier using curriculum learning strategies aligned with dialectal complexity and label cardinality. On the MLADI leaderboard, our best-performing LahjatBERT model achieves a macro F1 of 0.69, compared to 0.55 for the strongest previously reported system. Code and data are available at https://mohamedalaa9.github.io/lahjatbert/ . \setcode utf8 1 Introduction Arabic has a wide range of diverse dialects spoken across the Arab World. While Modern Standard Arabic (MSA) is used in official communication, education, and media, everyday conversations typically happen in local dialects 14 . Dialects vary between countries, and can even vary within same-country cities and local communities, creating a complex linguistic landscape 27 ; 8 . Arabic Dialect Identification (ADI) is the task that aims to identify the dialect of a sentence. ADI has been modeled as a single-label classification problem, with systems trained and evaluated on resources such as AOC 26 , MADAR 12 , QADI 1 , and NADI benchmarks 5 ; 6 ; 7 ; 3 . However, as illustrated in Table 1 , a single utterance may simultaneously sound natural to speakers from multiple countries, making it inherently multi-dialectal and difficult for traditional single-label classification methods to produce accurate results 19 ; 22 . Even moderately long sentences can be acceptable in multiple dialects 18 . Dialects Sentence Jordan, Palestine \RL اتفقنا نظل جنب بعض ع الحلوه والمره (We agreed to stick by each other through thick and thin.) Algeria, Egypt, Jordan, Palestine, Sudan, Syria, Tunisia, Yemen \RL احسبو حسابي معاكم (Make sure to count me in with you.) Algeria, Palestine, Sudan, Yemen \RL بسمتك يا زين تسوي ألف بسمه (Your smile, oh beautiful one, is worth a thousand other smiles.) All Arabic Dialects \RL اللهم الجنه لمن ذهبت أرواحهم إليك (O Allah, grant Paradise to those whose souls have returned to You.) Table 1: Examples illustrating dialect overlap in Arabic, sampled from the manually annotated NADI2024 Subtask 1 development set. Note: the dataset has labels for eight country-level dialects only. To better capture this phenomenon, the field has begun transitioning to Multi-Label Arabic Dialect Identification (MLADI), which allows an utterance to be tagged as acceptable in multiple dialects. 1 1 1 Multi-label Dialect Identification is being explored for other languages such as French and Spanish 11 ; 28 ; 21 . This transition has been driven by the evolution of the NADI shared tasks, a recurring benchmark series for Arabic Dialect Identification. While earlier editions of NADI (2020–2023) framed ADI as a single-label classification problem, assigning each sentence to a single country-level dialect 5 ; 6 ; 3 , the NADI2024 shared task introduced multi-label annotations to explicitly account for dialectal overlap 4 . However, in NADI2024 , multi-label annotations are provided only for the development and test sets, while the training data remains single-labeled, creating a mismatch between the nature of the task and the structure of the training data. As a result, MLADI introduces a weakly-supervised classification problem, where models must learn to predict multiple dialects based on datasets that provide only one ground-truth label per instance. In this work, we examine the mismatch between the multi-label nature of Arabic dialect usage and the single-label structure of existing ADI datasets by analyzing the behavior of binary dialect-specific acceptability classifiers trained on single-label NADI datasets 5 ; 6 ; 3 . Using the classifiers’ training dynamics 24 , we show that a substantial portion of samples treated as negative supervision is in fact judged acceptable by native speakers. This observation highlights the difficulty of accurately selecting negative samples when repurposing single-label, geo-located datasets for multi-label dialect identification and motivates explicit multi-label supervision. Building on these insights, we construct a multi-label training set by aggregating pseudo-labels from two heterogeneous sources: GPT-4o and a set of 18 binary dialect acceptability classifiers. Using this dataset, we train a BERT-based multi-label classifier, which we refer to as LahjatBERT . We train it under three settings: without curriculum learning, with an ALDi-aware curriculum, and with a label-cardinality-based curriculum ( 10 ) . The two curriculum strategies progressively expose the model to sentences of increasing dialectal ambiguity, allowing it to learn from simpler instances before confronting harder ones. Our contributions are threefold: 1. We provide an in-depth analysis of the limitations of reusing single-label ADI datasets for multi-label dialect acceptability. 2. We construct a pseudo-labeled multi-label dataset for MLADI by combining the predictions of GPT-4o and binary dialect classifiers. 2 2 2 Following the NADI shared-task license, we release only tweet IDs and derived labels, not the underlying tweet text. 3. We introduce LahjatBERT , a family of multi-label BERT-based models trained on the constructed dataset, and investigate curriculum-based training variants aligned with the multi-label structure of MLADI. The best-performing variant achieves 69.04\% macro-F1 on the MLADI leaderboard 18 , surpassing the top NADI2024 system and outperforming larger multilingual and Arabic-specific language models. 2 MLADI Task’s Setup and Previous Attempts Figure 1: Number of samples in each dialect after combining the NADI 2020, 2021, 2023 datasets. Samples with automatically estimated Arabic Level of Dialectness (ALDi; 17 ) \leq 0.11 are expected to be in MSA. The majority of the MSA samples are expected to be acceptable in all dialects. The MLADI dataset only provides a development set of 120 samples and a test set of 1000 samples. Each sample of the development set is manually labeled by annotators from 8 countries, while the test set has 11 country-level acceptability labels. To build multi-label ADI systems, the task requires using the following three single-labeled NADI datasets: NADI 2020 5 , NADI 2021 6 , and NADI 2023 3 . All these datasets provide tweets annotated with their estimated geo-location country label covering 18 countries: Algeria, Bahrain, Egypt, Iraq, Jordan, Kuwait, Lebanon, Libya, Morocco, Oman, Palestine, Qatar, Saudi Arabia, Sudan, Syria, Tunisia, the UAE, and Yemen. Since these datasets rely on the user geo-location rather than manual linguistic annotation, posts authored by users whose geo-location differs from their country of origin can be mislabeled 4 . NADI 2023 contains 1,000 tweets for each of 18 Arabic country-level dialects. By contrast, NADI 2020 and NADI 2021 are more imbalanced, with dialects such as Bahraini and Qatari underrepresented relative to more frequent varieties like Egyptian and Iraqi. The combined distribution of samples across dialects is shown in Figure 1 . Previous MLADI Attempts. 15 used a nearest-neighbour approach to predict multiple labels for each sample, by encoding the training data samples and the test set into the same embedding space. 16 tackled the weak supervision problem by applying a similarity-based label expansion strategy. Their method, SIMMT, heuristically assigns additional labels to each sample based on vocabulary similarity between dialects, followed by multi-label fine-tuning of transformer models. This approach demonstrated that moving from single-label to multi-label supervision can improve performance on the MLADI task. Our work replaces heuristic label expansion with a pseudo-labeling framework that integrates complementary signals from multiple models to produce multi-label annotations. This yields a richer supervision signal than surface-level similarity measures. 3 Difficulties of Using Existing Datasets for Dialect Acceptability Classification An intuitive idea to build multi-label ADI systems is to repurpose single-label ADI datasets for training multiple independent binary classifiers, each of which assesses the acceptability of a sentence in a specific dialect. For a specific country-level dialect ( Cntry ), one might take samples geolocated to Cntry as positive (i.e., acceptable ) samples for the country’s classifier, and samples geolocated to other countries as negatives . Previous attempts have shown that this technique does not achieve the best classification performance 16 ; 15 . However, they have not analyzed the reasons for the failure of this technique, which we investigate here. 3.1 Other Countries’ Samples Are Not Always Negative Samples Definition For a dialect acceptability model, the negative class represents sentences with any linguistic feature (e.g., a morpheme or a lexical item) that is not acceptable in the considered dialect . The positive class represents the remaining sentences acceptable in this dialect or in MSA. The majority of the samples geo-located to the considered dialect are expected to be positive samples. The fact that some of these samples could also be acceptable in other dialects does not impact their categorization as positive samples, according to the aforementioned definition. To the contrary, considering samples geo-located to other dialects as negative samples is problematic. More specifically, a subset of these samples is expected to also be acceptable in the dialect considered. Consequently, this subset of negative sentences should be reassigned to the positive class. We next show how model training dynamics could guide the identification of wrongly assigned negative samples. 3 3 3 4 found that NADI’s geo-location method has a moderate to high accuracy of ensuring a sentence’s acceptability in a specific country-level dialect. Figure 2: The training dynamics for 6 binary acceptability classifiers, characterized by the mean confidence in the label across different steps/stages of the model training (y-axis), and the standard deviation of these confidence values (x-axis). Each pair shows the training dynamics’ metrics for the non-MSA positive (left) and negative (right) samples of a single classifier, with the respective number of samples shown above each subplot. Note: Sample’s correctness ranges are : 0 : ]0, 0.2[ : [0.2, 0.4[ : [0.4, 0.6[ : [0.6, 0.8[ : [0.8, 1[ : 1 3.2 Training Dynamics and Multi-label Samples 24 used three metrics tracked during the model training process to categorize each sample’s difficulty to be learned. For a training sample x i with a label y i , the confidence that the model assigns to the target y i is tracked at the end of each training epoch (or every number ‘ N ’ of training batches). The mean and standard deviation of the tracked confidence scores assigned to the sample’s target label are termed the Confidence and Variability metrics, respectively. Correctness represents the percentage of epochs (or N batches) for which the model assigns a higher probability to the sample’s target label than all the other labels. Based on these metrics, the training dataset is split into three main categories— (1) easy to learn samples: high- Confidence and low- Variability , (2) ambiguous samples: moderate- Confidence and high- Variability , and (3) hard to learn : low- Confidence and low- Variability . Intuition A dialect acceptability classifier would struggle to learn negative samples that are also acceptable in the considered dialect, as these samples should belong to the positive class. These samples are expected to be ambiguous or even hard to learn . Methodology We train 18 dialect acceptability classifiers, one for each country represented in the NADI datasets. For each country’s classifier, samples of the NADI datasets geo-located to this country, and MSA samples (ones with ALDi < 0.11) are considered as positive (acceptable) samples, with the remaining samples considered as negative (unacceptable) samples. Confidence scores are tracked every 300 steps ( \approx\frac{1}{5} th of an epoch) for 5 epochs. However, the first epoch’s confidence scores are ignored, as the model’s training dynamics could be unstable during the early learning stages 24 . Findings Figure 2 shows the training dynamics for six different acceptability classifiers, for the positive (non-MSA) samples and the negative samples. First, a few samples (<50) for each set have a correctness value of 0. Moreover, the larger the imbalance between the number of positive and negative samples becomes, the smaller the percentage of negative samples with non-perfect correctness is. For instance, 7.2% (n=3,046) of Egypt’s negative samples have non-perfect correctness scores compared to only 1.9% (n=920) for Sudan. These negative samples with non-perfect correctness scores are potentially wrongly assigned samples. 3.3 Usability of Training Dynamics in Flagging Wrongly Assigned Samples To understand the effectiveness of training dynamics in identifying wrongly assigned samples, we engaged native speakers to manually evaluate the acceptability of sentences with varying correctness scores. Specifically, we recruited one annotator from the following countries: Egypt (EG), Iraq (IQ), Algeria (DZ), Tunisia (TN), Jordan (JO), and Sudan (SD). 4 4 4 Notably, the six recruited annotators previously participated in the annotation of NADI 2024’s evaluation sets. For each country, we sample 10 examples from each of the seven correctness ranges: 0, ]0, 0.2[, [0.2, 0.4[, [0.4, 0.6], [0.6, 0.8[, [0.8, 1[, 1 . In total, 140 samples are annotated for each dialect (70 of which are positive and the remaining 70 are negative), unless one of the bins had fewer than 10 samples. The annotators follow the same guidelines of NADI 2024’s first subtask 4 , whereby they answer: Is it possible that the tweet is authored by someone who speaks one of your country’s dialects? Options: Yes, Not Sure/Maybe, or No . Results Inspecting the number of acceptable samples in each bin in Table 2 , a large number of negative samples with correctness <1 are rated as acceptable, indicating the effectiveness of using non-perfect correctness scores to flag wrongly assigned negative samples. Conversely, a majority of the positive samples are rated as acceptable for the different correctness bins, even for correctness scores that are less than 0.6, which could be attributed to the class imbalance toward negative samples. Label Correctness 0 ]0,0.2[ [0.2,0.4[ [0.4,0.6[ [0.6,0.8[ [0.8,1[ 1 EG +ve 2 6 6 6 7 9 10 -ve 10 10 10 9 8 4 2 IQ +ve 4 2 1 6 4 5 7 -ve 9 6/8 10 9 7 5 2 DZ +ve 3 4 5 5 8 7 9 -ve 9 5/5 8 9 9 8 4 TN +ve 1 3 1 0 1 5 7 -ve 2/2 4/4 8/9 7 5 4 1 JO +ve 3 7 6 5 8 10 9 -ve 2/2 5/5 10 10 9 9 2 SD +ve 5/8 1/2 6 7 10 8 10 -ve 5/5 3/3 8/8 7 8 10 1 Table 2: The number of acceptable samples for the different correctness bins. For each row, each bin contains 10 total samples, except for a few bins marked by the acceptable/total format. Note: Values marked in blue indicate that >50% of the bin’s samples are acceptable, while values marked in red indicate that <50% of the bin’s samples are acceptable. The high-correctness bins of the positive class are expected to have a large number of acceptable samples. Conversely, the high-correctness bins of the negative class are expected to have a small number of acceptable samples, which is not the case. Country codes follow the ISO 3166-1 alpha-2 standard. Moving Forward Our analysis shows that a large proportion of negative samples with non-perfect correctness scores should be reassigned to the positive class. However, manual annotation is still required to assess these automatically flagged samples. Moreover, the class imbalance seems to have an impact on the model’s dynamics. Future work could consider using a human and model in the loop setup 25 . More specifically, the initial model’s training dynamics are used to automatically identify ambiguous samples, which are then manually reassigned to the correct class. Afterward, another model is trained from scratch on the dataset after reassignment, with the new model’s dynamics used to automatically identify new ambiguous samples. 4 Multi-Label ADI Dataset Creation It seems inevitable that building multi-label ADI models requires the presence of multi-label ADI training datasets, especially to reduce the number of false negatives for each dialect (i.e., samples wrongly assumed unacceptable in that dialect). However, building large enough multi-label ADI datasets is expensive, since annotating just 1,120 samples by speakers of 9 countries could cost as much as $1,700 4 . Hence, we propose two pseudo-labeling methods for building multi-label ADI datasets: (1) Binary Dialect Classifiers. We again build 18 independent acceptability classifiers, one per country-level dialect dia , each predicting whether a sentence x is acceptable in dialect dia . These classifiers are trained only on the balanced NADI 2023 dataset to avoid skewed pseudo-labels. For a sentence x_{i} , we generate 18 acceptability pseudo-labels using the 18 classifiers as: \hat{\mathbf{y}}^{\mathrm{BIN}}_{x_{i}}=\left(Accept_{dia_{1}}(x_{i}),\dots,Accept_{dia_{18}}(x_{i})\right). Here, we notably rely on the Arabic Level of Dialectness ( 17 , ALDi;) score as a global signal characterizing the degree of dialectness of each sentence ( a_{i}~=~\mathrm{ALDi}(x_{i})\in[0,1] ). 18 showed that ALDi moderately correlates with the number of dialects in which a sentence is acceptable. The higher the ALDi score of a sentence, the less the number of dialects in which the sentence is acceptable. For the positive samples of dia ’s acceptability classifier Accept_{dia}(x_{i}) , we consider all sentences from the NADI datasets that are geolocated to dia , in addition to MSA sentences with a_{i}<0.11 , which are broadly acceptable across dialects, irrespective of their geolocation. Since treating all sentences geolocated to other regions as negative examples leads to systematic errors (as shown in Section 3 ), we prioritize the precision of selecting true negative samples . To this end, we only select sentences that are (1) geolocated to non-neighbouring dialect regions, (2) with high dialectness ( a_{i}>0.77 ), where linguistic overlap with dialect dia is less likely. 5 5 5 See Appendices B , C and D for more implementation details. (2) GPT-Based Multi-Label Annotation. For each sentence, we obtain multi-label annotations from GPT-4o by prompting the model to independently assess the acceptability of the sentence in each of the 18 dialects. This yields a binary relevance vector \hat{\mathbf{y}}^{\mathrm{GPT}}_{i}\in\{0,1\}^{18}. The full prompting template is provided in Appendix E . 4.1 Quality of Supervision Signals To evaluate the two labeling methods, we use the development set of the NADI2024 shared task 4 , which has 120 sentences annotated for 8 country-level dialects. This provides a useful benchmark for evaluating the performance of the two methods, as shown in Table 3 . Pseudo-Labeling Method P macro R macro F1 macro Acc. (1) Binary Dialect Classifiers 78.2 51.4 60.4 76.2 (2) GPT-4o Pseudo-Labels 73.5 66.3 67.8 77.4 Hybrid ALDi-Based Labels (§ 4.2 ) 77.5 62.8 68.5 79.3 Table 3: Macro-averaged Precision, Recall, F1, and Accuracy of the two methods in addition to the aggregation method (§ 4.2 ) on the NADI2024 development set. Our conservative way of selecting the samples for the binary classifiers results in a high macro precision of 78.2 and a moderate macro recall of 51.4. In contrast, GPT-4o achieves substantially higher recall and a better overall F1 score. This hints that GPT-4o is more willing to assign multiple dialect labels, particularly in cases where dialectal overlap is present. 4.2 Pseudo-Labels Aggregation To better understand the results in Table 3 , we analyze performance across ALDi ranges in Table 4 . For MSA samples ( [0,0.11) ), both GPT-4o and the Binary Classifiers achieve perfect precision. For highly dialectal samples ( [0.77,1] ), the Binary Classifiers attain higher precision, indicating more reliable negative supervision when strong dialectal cues are present. In contrast, GPT-4o consistently achieves higher recall and F1 in the intermediate ALDi ranges ( [0.11,0.77] ), where dialectal overlap is more common. This behavior is consistent with the conservative construction of the Binary Classifiers, whose negative samples are restricted to highly dialectal cases ( a_{i}>0.77 ), improving precision at the extremes but limiting coverage in the mid-range. ALDi Bin Supervision Source P macro R macro F1 macro [0,0.11) (1) Binary Classifiers 100.00 94.64 97.12 (n=7) (2) GPT-4o 100.00 90.48 94.84 [0.11,0.44) (1) Binary Classifiers 76.99 46.81 55.99 (n=16) (2) GPT-4o 84.72 61.36 69.72 [0.44,0.77) (1) Binary Classifiers 63.28 36.72 44.38 (n=48) (2) GPT-4o 63.86 56.94 57.15 [0.77,1.0] (1) Binary Classifiers 77.60 51.38 59.03 (n=49) (2) GPT-4o 62.36 68.14 62.62 Table 4: Performance of the two pseudo-labeling methods on subsets of the NADI2024 development set. Consequently, we aggregate the predictions of the two pseudo-labeling methods by using the Binary Classifiers at the ALDi extremes ( a_{i}<0.11 or a_{i}>0.77 ) and GPT-4o in the intermediate range ( 0.11\leq a_{i}\leq 0.77 ). This aggregation combines the strengths of both methods and achieves the best macro F1 and accuracy on the evaluation set (last row of Table 3 ), reflecting an improved precision–recall trade-off. Figure 3: Number of samples for each label cardinality according to the three pseudo-labeling methods. 5 Multi-Label Dialect Classification Models Using the pseudo-labeled multi-label ADI dataset described in Section 4 , we fine-tune MARBERT 2 as a multi-label classifier over the 18 Arabic dialects (country-level ADI). We refer to the resulting family of models fine-tuned under different settings, as LahjatBERT . We train using binary cross-entropy with logits loss, treating each label independently. During fine-tuning, we freeze the bottom 8 transformer layers of MARBERT and update only the top 4 layers and the classification head. We do not fully fine-tune all MARBERT layers since fine-tuning only a fraction of the final layers recovers most downstream effectiveness 20 . Also, we discard zero-cardinality samples (i.e., instances for which the pseudo-labeling step assigns no dialect), since keeping them would treat the samples as negative for all 18 dialects and inject systematic noise. We report the complete training and inference hyperparameters in Appendix A . An analysis of the dataset ( Figure 3 ) reveals a strong skew in the cardinality distribution: most samples contain only one or two active dialects. This bias might encourage the model to predict low-cardinality outputs, substantially reducing recall for sentences acceptable in multiple dialects. To mitigate the dataset’s inherent low-cardinality bias, we adopt two strategies for curriculum learning (CL). In both strategies, the complexity of the training examples is increased in a controlled manner during the model training process, to achieve better model generalization 10 . The exact ordering of both the cardinality buckets and the ALDi buckets is motivated in Appendix F . 5.1 Cardinality-Based Curriculum Learning In this strategy, the samples’ label cardinality is used as a proxy for their respective complexities. For a sentence x_{i} with a pseudo-labeled target vector \mathbf{y}_{i}\in\{0,1\}^{18} , the label cardinality c(x_{i})~=~\|\mathbf{y}_{i}\|_{0} represents the number of dialects in which the sentence is acceptable. This strategy is proposed to mitigate the skewness of the training dataset samples toward lower cardinalities. We partition the training set into cardinality buckets B_{c} , where B_{c} contains samples with label cardinality c . We then define a curriculum ordering over these buckets using the mean loss of each cardinality. Let \pi(e) denote the cardinality index of the bucket selected at curriculum stage e . At stage e , the model is trained on (1) all examples from B_{\pi(e)} , and (2) an equal number of randomly sampled examples from each previously introduced bucket B_{\pi(1)},\dots,B_{\pi(e-1)} . The model first trains exclusively on the bucket introduced at the first curriculum stage. In the next stage, the bucket selected next is introduced together with an equal number of samples drawn from each previously introduced bucket. This progression continues until the full cardinality range in the dataset is incorporated. The overall training schedule is illustrated in Figure 4 , which visualizes how buckets are gradually introduced while maintaining balanced exposure across earlier stages. Figure 4: Illustration of the cardinality-based curriculum schedule, showing the progressive introduction of higher difficulty cardinality samples. The numerical values are illustrative and do not reflect the actual dataset. 5.2 ALDi-Based Curriculum Learning ALDi provides another intuitive proxy for the samples’ complexities. Samples with intermediate ALDi scores are expected to be harder than MSA samples with low scores and high-score samples that show clear cues of a specific dialect. We divide the continuous ALDi range into four contiguous intervals: I_{1}~=[0,~0.11),I_{2}~=[0.11,~0.44),I_{3}~=[0.44,~0.77),I_{4}~=[0.77,~1]. For each interval I_{k} , we construct a bucket B_{k} that contains all training examples whose ALDi score falls in that range: B_{k}=\{x_{i}\mid a(x_{i})\in I_{k}\} ; where a(x_{i})~\in~[0,1] denotes sentence x_{i} ’s ALDi score. We define a curriculum ordering over the ALDi buckets based on average training loss. Let \pi(e) denote the index of the ALDi bucket selected at epoch e , with buckets ordered from lowest to highest loss. At epoch e , the model is trained on (1) all examples from the current bucket B_{\pi(e)} , and (2) a random subset of examples from each previously introduced bucket B_{\pi(1)},\dots,B_{\pi(e-1)} , following the same sampling strategy as the cardinality-based CL. 6 Results We evaluate our multi-label dataset construction and training strategies on the NADI2024 development set, which provides multi-label annotations for 8 country-level dialects . Since our training and supervision span 18 dialects , we report metrics only on the overlapping label set. 6.1 Multi-Label Model Performance We evaluate how a single multi-label classifier can learn from different supervision signals by training the same model architecture on each dataset without curriculum learning. This setup isolates the effect of the supervision labels themselves. As shown in Table 5 (Block II), the model trained on GPT-4o labels achieves higher macro Recall but lower Precision, indicating a tendency to over-predict dialect labels. In contrast, training on the hybrid labels yields higher Precision and Accuracy while maintaining comparable macro F1. This behavior is consistent with the design of the hybrid labels, which combines the conservative behavior of the binary classifiers at ALDi extremes with the richer GPT-based labels in the ALDi mid-range. Model P macro R macro F1 macro Acc. (I) Baseline NADI 2024 Baseline 71.2 30.9 39.7 69.3 (II) Multi-Label Model Performance Trained on GPT-4o labels 69.6 65.7 66.2 75.5 Trained on hybrid labels 73.7 63.7 67.4 77.5 (III) Curriculum Learning on Hybrid Dataset Hybrid + Cardinality-Based CL 69.0 80.6 72.7 77.5 Hybrid + ALDi-Based CL 71.4 71.3 70.3 78.2 Table 5: Macro-averaged Precision, Recall, F1, and Accuracy on the NADI2024 development set. Results are grouped into (i) multi-label model performance without CL, and (ii) the effect of CL. 6.2 Impact of Curriculum Learning We next examine the effect of curriculum learning by comparing curriculum-based training with the baseline setting without a curriculum. As shown in Table 5 (Block III), both curriculum learning strategies improve macro F1 relative to training without curriculum learning, while consistently increasing recall and reducing precision. In our setup, the curriculum order is derived from a loss-based criterion (Appendix § F ), where examples introduced at the latest stages tend to be more ambiguous, corresponding to intermediate label cardinalities and intermediate ALDi scores. One possible interpretation of the observed recall increase is that exposure at the latest curriculum stages to higher-loss, more ambiguous examples encourages the model to activate a larger set of labels per instance. These examples are characterized by intermediate ALDi scores and multiple valid dialect labels per instance. This broader label coverage may help recover more relevant dialect labels, leading to higher recall. At the same time, predicting more labels per instance can make the model less conservative, which is reflected in the accompanying reduction in precision. This interaction also helps explain why curriculum learning is most effective when applied to the hybrid supervision. Since the hybrid annotations exhibit higher precision than GPT-4o labels alone, they help limit the precision loss associated with increased recall. 6.3 Generalization on the MLADI Test Set We evaluate whether the improvements on the development set generalize to the MLADI test set. Table 6 reports the performance of the three LahjatBERT variants compared with the NADI 2024 baseline and previously reported systems. The NADI 2024 baseline is a single-label dialect identification model, but it is converted into a multi-label predictor at test time. For each sentence, it computes a softmax distribution over the 18 dialect classes, then selects the most likely dialects until their cumulative probability reaches a fixed value (we use the Top-90% setting, i.e., P{=}0.9 ), and returns those dialects as the predicted label set 4 . All LahjatBERT variants outperform the shared-task baseline and prior approaches in terms of macro F1, indicating that the gains obtained from the constructed supervision and training strategies extend beyond the development setting. Among the LahjatBERT variants, the ALDi-based curriculum achieves the highest macro F1, while the cardinality-based curriculum yields the highest recall, mirroring the precision-recall trade-offs observed on the development set. 6 6 6 We analyze the models’ predictions in the Appendix (§ G ). Model P macro R macro F1 macro Acc. LahjatBERT (no curriculum) 69.0 69.7 68.0 79.1 LahjatBERT + Cardinality CL 59.3 81.0 66.6 73.9 LahjatBERT + ALDi CL 65.0 76.4 69.0 78.2 Aya-32B 13 49.5 64.5 54.5 65.6 Elyadata 16 50.2 56.9 52.4 67.0 NADI 2024 Baseline 64.8 39.9 47.0 72.3 Table 6: Macro-averaged Precision, Recall, F1, and Accuracy on the MLADI test set 18 , which contains 1,000 sentences annotated for 11 dialects. 7 Conclusion and Future Work This work examines the limitations of reusing single-label Arabic dialect identification (ADI) datasets for multi-label ADI (MLADI). We specifically highlight that the difficulty of building binary dialect acceptability classifiers lies in the selection of negative samples. While training dynamics can help in automatically flagging wrongly-assigned negative samples, manual verification is still required to assess the acceptability of these flagged samples. Using single-label ADI data, we construct a pseudo-labeled multi-label dataset by aggregating predictions from GPT-4o and binary dialect acceptability classifiers, and introduce LahjatBERT , a family of BERT-based multi-label models that outperform existing MLADI systems. Future work could incorporate targeted human annotation for samples identified as ambiguous by the analysis, enabling iterative refinement of multi-label supervision. Finally, our CL approach substantially improves recall. Future work could examine whether similar gains extend to other multi-label tasks beyond dialect identification. Limitations Our evaluation is limited by the label coverage of available benchmarks. Although our training and supervision signals span 18 dialects, the NADI2024 development set provides multi-label annotations for only 8 country-level dialects, so development metrics are computed only on the overlapping label subset. Likewise, the test set covers only 11 dialects, restricting conclusions about generalization to the full 18-dialect label space. A second limitation originates from the fact that NADI relies on user geo-location as a proxy for dialect rather than direct linguistic annotation, which can be noisy when posting location and actual dialect do not align. Ethics and Broader Impact This work uses anonymized Arabic tweets from the publicly released NADI shared-task datasets, collected via the Twitter API under the platform’s terms. Following NADI licensing restrictions, we release only tweet IDs and our automatically generated multi-label annotations (not the underlying tweet text), along with code and trained models to enable reproducibility. The research is methodological, and while dialect identification can support beneficial language technologies and sociolinguistic analysis, it also carries risks of misuse (e.g., profiling or surveillance), which we explicitly do not endorse and is outside the scope of this work. Appendix A Experimental Details We provide the full training and inference hyperparameters for the multi-label setup described in section 5 . We fine-tune a BERT-based model for multi-label country-level Arabic dialect identification with 18 binary labels, where the classification head outputs one logit per country. Training minimizes binary cross-entropy with logits, which implements a combination of sigmoid and binary cross-entropy and treats each label independently. We split the data into 90% training and 10% validation with a fixed random seed of 42. During fine-tuning, we freeze the bottom 8 transformer layers of MARBERT and update only the top 4 layers and the classification head. We set both the hidden-state and attention dropout probabilities to 0.3. We train for 3 epochs with a batch size of 24 for both training and evaluation, evaluating once per epoch. The best checkpoint is selected based on validation micro F1. At inference time, we apply a sigmoid to the logits and use a threshold of 0.3 to obtain binary label assignments. We chose this value by maximizing validation micro F1 on the held-out split, the default threshold of 0.5 yields lower validation macro F1, so we use 0.3 for all reported results. Appendix B ALDi Threshold Selection We adopt two ALDi thresholds, a_{i}=0.11 and a_{i}=0.77 , to distinguish sentences with minimal dialectal evidence from those with strong dialectal evidence. These thresholds are grounded in the annotation scheme and definition of the Arabic Level of Dialectness (ALDi) score introduced by 17 . ALDi Annotation Scheme. Following the ALDi annotation protocol described by 17 , each sentence is annotated by three native Arabic speakers. Annotators assign one of four ordinal labels reflecting the degree of dialectness: MSA (0), Little ( \tfrac{1}{3} ), Mixed ( \tfrac{2}{3} ), and Most (1). The ALDi score for a sentence is defined as the mean of the three annotations. Consequently, ALDi values lie in [0,1] and take discrete steps of \tfrac{1}{9} . Derivation of the Thresholds. Under this definition, the smallest non-zero ALDi value is \frac{0+0+\tfrac{1}{3}}{3}=\tfrac{1}{9}\approx 0.11, which corresponds to the weakest possible evidence of dialectal content, where two annotators label the sentence as MSA and one assigns a Little dialect label. Conversely, an ALDi score of \frac{1+1+\tfrac{1}{3}}{3}=\tfrac{7}{9}\approx 0.77 corresponds to strong dialectal evidence, where two annotators assign the highest dialectness label ( Most ) and the third assigns at least Little . These two values therefore mark natural boundaries between minimal dialectal signal ( a_{i}<0.11 ), strong dialectal signal ( a_{i}>0.77 ), and an intermediate range characterized by mixed or graded annotator judgments. ALDi Score Distribution. ALDi scores are concentrated near the MSA ( a_{i}<0.11 ) and highly dialectal ( a_{i}>0.77 ) ranges, with fewer samples in the mid-range ( 0.11\leq a_{i}\leq 0.77 ), as shown in Figure B1 . Figure B1: Distribution of ALDi scores in the dataset. Vertical dashed lines indicate the thresholds distinguishing MSA text ( a_{i}<0.11 ), low–medium dialectness ( 0.11 – 0.44 ), medium dialectness ( 0.44 – 0.77 ), and highly dialectal text ( a_{i}>0.77 ). Appendix C Non-Neighbouring Dialect Regions When constructing negative samples for the binary dialect classifiers ( section 4 ), we restrict negatives to sentences geolocated in non-neighbouring countries. Neighbouring countries are defined as those sharing a land border with the target country, and negative samples are drawn exclusively from non-bordering countries. By restricting negative samples to non-bordering countries, we avoid cases where sentences may be linguistically compatible with the target dialect, resulting in more reliable negative supervision. Appendix D Dataset Diagnostics This appendix presents diagnostics of the constructed multi-label dataset, focusing on how label cardinality varies with the Arabic Level of Dialectness (ALDi) score for the supervision sources considered in this work. D.1 Label Cardinality Across ALDi Ranges Figure D2 shows the distribution of label cardinality across ALDi categories for each labeling method, as well as for the aggregated dataset. Binary Dialect Classifiers. The binary classifiers exhibit a strong dependence on the ALDi score. For sentences with low ALDi values (MSA text), they frequently activate many dialect labels, resulting in high cardinality. For sentences with high ALDi values (strongly dialectal text), they tend to activate very few labels. In the intermediate ALDi ranges ( 0.11\leq a_{i}\leq 0.77 ), where explicit negative supervision is absent, classifier outputs concentrate at either high or low cardinalities, and intermediate label counts are rarely observed. GPT-4o. GPT-4o displays a smoother relationship between ALDi and label cardinality. While cardinality generally decreases as ALDi increases, GPT-4o assigns moderate numbers of labels more frequently in the Low–Med and Medium ALDi ranges. This pattern is consistent with graded representations of dialectal overlap and contrasts with the polarized outputs of the binary classifiers in the same ALDi regions. Aggregation Effects. The aggregated dataset (right panel of Figure D2 ) combines these two behaviors. At low and high ALDi values, where binary classifier predictions are stable, the aggregated cardinality closely matches their outputs. In the intermediate ALDi ranges, where binary classifier predictions are concentrated at extreme cardinalities, the aggregation relies on GPT-4o, resulting in intermediate label counts and a smoother dependence of cardinality on ALDi. Figure D2: Cardinality across ALDi categories for the three labeling methods. Due to instabilities, we rely on GPT-4o for ambiguous cases and on the binary classifiers when the dialectness is clearly low (MSA) or high. This leads to more consistent and linguistically plausible multi-label patterns. D.2 Overall Label Cardinality Distribution Figure 3 shows the overall distribution of label cardinality for the supervision sources used in this work, independent of ALDi conditioning. Appendix E Prompt Template for GPT-Based Annotation Instruction: You are a native Arabic speaker and highly qualified linguist with expert-level understanding of regional Arabic dialects. Given the sentence provided, evaluate its dialectal characteristics independently for each of the following dialects: Iraq, Egypt, Morocco, Libya, UAE, Saudi Arabia, Bahrain, Syria, Lebanon, Oman, Palestine, Algeria, Jordan, Tunisia, Kuwait, Yemen, Sudan, and Qatar. Return findings in JSON format: { "Iraq": 0/1, "Egypt": 0/1, ... "Qatar": 0/1 } Input sentence: {tweet} Figure E3: GPT-based annotation prompt template. Appendix F Curriculum Learning Ordering To obtain a principled ordering for both the cardinality-based curriculum learning and the ALDi-based curriculum learning, we first train a baseline MARBERT model on the aggregated multi-label dataset without any curriculum learning. After fine-tuning, we freeze this baseline model and run inference on the full training set, recording the per-example loss. We then aggregate these losses in two ways: (i) by label cardinality, computing the mean loss for each cardinality value, and (ii) by ALDi bin, also computing the mean loss but for each ALDi interval. The resulting mean-loss profiles are visualized in Figure F4 (mean loss per cardinality) and Figure F5 (mean loss per ALDi bin). We treat the mean loss of a bucket as a proxy for its difficulty, and order the stages from lower-loss (easier) buckets to higher-loss (harder) buckets. This difficulty-based ordering directly determines the progression of stages in both the cardinality-based and ALDi-based curriculum learning schedules described in section 5 . Figure F4: Sorted mean loss per cardinality, to measure the difficulty for the baseline model in predicting different cardinalities. Figure F5: Sorted mean loss per ALDi bin, to measure the difficulty for the baseline model in predicting different cardinalities. Appendix G Model’s Predictions on MADAR’s Samples To analyze the behavior of the three newly introduced models, we study their predictions on 200 sentences from the MADAR-26 corpus 23 , for each of the 6 anchor dialects identified by the corpus creators. Additionally, we contrast their predictions to two baseline models: a single-label DI model DID-Country , and the multi-label DI NADI2024 shared task’s baseline . Comparing our three LahjatBERT models to the NADI2024 baseline, it is clear that the baseline is more susceptible to predicting the sentence to be acceptable in countries from different regions than our newly introduced model. For instance, out of the 200 sentences in Rabat’s dialect (spoken in Morocco), the baseline model unintuitively predicts that 190 of them are acceptable in Egypt, in comparison to less than 20 of them predicted to be acceptable by the three LahjatBERT models. Hence, assuming that the labels within the top-p (p=90%) of a single-label DI model as acceptable dialects for the input sentence is not an optimal strategy for multi-label dialect identification. For the three LahjatBERT models, the Cardinality-based one seems to frequently predict labels from other regions, which explains its lower precision yet higher recall than the two other models. The ALDi-based model seem to be achieving the best compromise between the precision and the recall, as indicated by the fact it achieves the highest overall macro-F1 score. Dialect DID-Country NADI2024 Baseline LahjatBERT LahjatBERT + LahjatBERT + 23 4 ALDi CL Cardinality CL BEI (200) LB 129 , SY 32 , JO 15 , PS 12 , EG 4, SA 3, IQ 2, OM 2, MA 1 LB 200 , SY 200 , JO 199 , PS 199 , EG 180, IQ 174, TN 160, SD 127, BH 86, MA 78, KW 68, DZ 57, SA 56, QA 53, LY 41, OM 36, YE 25, AE 19 SY 165 , LB 149 , PS 131 , JO 121 , IQ 36, SA 35, AE 34, KW 32, BH 31, OM 31, QA 29, YE 27, EG 23, LY 18, SD 17, TN 10, DZ 9, MA 9 SY 174 , LB 146 , PS 135 , JO 129 , OM 35, SA 33, AE 33, IQ 31, KW 31, BH 30, QA 30, YE 26, EG 25, SD 19, LY 18, TN 9, DZ 8, MA 7 SY 188 , LB 184 , PS 181 , JO 173 , SA 45, AE 44, BH 42, OM 42, KW 41, QA 41, YE 41, IQ 35, EG 31, SD 23, LY 22, MA 9, TN 9, DZ 7 CAI (200) EG 147 , SA 12, SY 12, SD 9 , JO 7, DZ 5, LY 4, TN 4, MA 3, YE 3, IQ 2, PS 2, QA 2, KW 1, LB 1, BH 1, OM 1, AE 1 EG 200 , YE 193, SA 192, SD 192 , PS 188, JO 187, LY 182, LB 178, SY 167, MA 159, OM 142, TN 130, IQ 73, KW 54, QA 51, AE 40, DZ 20 EG 190 , PS 131, SD 115 , LB 54, SA 54, JO 52, SY 52, LY 51, KW 46, IQ 44, BH 44, QA 44, AE 43, OM 42, YE 42, TN 14, MA 12, DZ 10 EG 188 , PS 152, SD 132 , LY 49, SA 49, JO 46, LB 45, IQ 44, SY 44, KW 43, BH 41, OM 41, QA 41, AE 41, YE 41, TN 11, MA 9, DZ 8 PS 193, EG 189 , SD 165 , SA 135, LB 103, JO 99, SY 93, QA 83, OM 81, AE 80, BH 79, YE 79, KW 74, LY 73, IQ 60, TN 14, MA 12, DZ 10 DOH (200) QA 144 , SA 17 , IQ 7, JO 5, SY 5, OM 4 , MA 3, SD 3, TN 3, PS 3, LY 2, EG 2, YE 2 SA 200 , QA 199 , OM 198 , LY 194, JO 191, SY 181, IQ 163, SD 156, YE 137, AE 122 , MA 112, KW 86 , LB 80, TN 77, PS 76, BH 64 , EG 59, DZ 14 SA 173 , KW 154 , BH 142 , AE 136 , QA 133 , OM 128 , IQ 115, YE 111, PS 93, JO 91, SY 91, LB 89, SD 55, LY 54, EG 52, MA 24, TN 24, DZ 21 SA 179 , KW 165 , BH 151 , QA 147 , AE 142 , OM 134 , YE 117, IQ 111, PS 105, LB 102, JO 100, SY 100, EG 53, SD 49, LY 46, MA 17, TN 17, DZ 14 SA 197 , BH 194 , QA 193 , KW 190 , AE 190 , OM 175 , YE 149, IQ 132, PS 115, JO 111, LB 96, SY 96, SD 53, EG 50, LY 46, MA 14, TN 13, DZ 11 RAB (200) MA 176 , DZ 6 , TN 4 , JO 3, LY 3 , SA 3, PS 2, OM 1, SY 1, QA 1 MA 198 , DZ 196 , LY 194 , EG 190, QA 188, LB 180, AE 165, SA 159, SY 154, SD 151, TN 151 , PS 147, KW 143, BH 125, IQ 70, JO 41, OM 32, YE 11 DZ 172 , MA 171 , TN 84 , LY 27 , EG 16, PS 15, IQ 13, SY 13, JO 11, LB 11, SA 11, OM 10, AE 10, KW 9, BH 8, QA 8, YE 8, SD 7 DZ 181 , MA 175 , TN 68 , LY 23 , PS 14, EG 10, IQ 8, SY 8, JO 7, SA 7, LB 6, AE 6, QA 6, KW 5, BH 5, OM 5, YE 4, SD 1 MA 178 , DZ 174 , TN 130 , LY 52 , PS 32, SY 30, JO 29, LB 27, AE 25, KW 24, OM 24, SA 24, QA 24, YE 24, IQ 21, BH 20, EG 19, SD 16 TUN (200) TN 167 , DZ 6 , SA 5, JO 4, IQ 3, OM 3, PS 3, LY 2 , MA 2 , SY 2, QA 2, YE 1 LY 198 , TN 197 , LB 192, IQ 191, DZ 190 , EG 189, MA 183 , OM 181, SD 180, QA 160, YE 150, KW 104, PS 99, AE 60, SY 42, SA 40, JO 28, BH 14 TN 159 , LY 149 , DZ 143 , MA 94 , IQ 24, SY 24, PS 24, LB 23, JO 21, EG 19, KW 17, SA 17, AE 16, YE 15, BH 14, OM 14, SD 14, QA 14 TN 163 , DZ 155 , LY 153 , MA 57 , PS 28, IQ 25, JO 23, LB 23, SY 23, EG 21, KW 20, SA 20, AE 20, OM 19, YE 18, BH 17, QA 16, SD 15 LY 170 , TN 162 , DZ 159 , MA 103 , PS 48, SY 44, JO 42, LB 41, IQ 35, SA 33, KW 32, OM 31, AE 31, QA 30, YE 30, BH 29, EG 28, SD 21 MSA (200) OM 162 , SA 162 , SD 152 , LY 151 , SY 151 , DZ 150 , IQ 149 , JO 149 , EG 149 , PS 148 , KW 147 , LB 147 , BH 147 , MA 147 , QA 147 , TN 147 , AE 147 , YE 147 OM 200 , SA 200 , SD 200 , IQ 195 , YE 189 , JO 165 , AE 151 , QA 145 , LY 144 , EG 143 , DZ 111 , KW 100 , BH 91 , TN 75 , MA 57 , SY 56 , PS 30 , LB 22 IQ 196 , JO 195 , LB 195 , PS 195 , SA 194 , SY 194 , AE 194 , KW 193 , LY 193 , EG 193 , OM 192 , YE 192 , BH 191 , SD 191 , QA 190 , TN 188 , MA 187 , DZ 183 IQ 198 , EG 198 , LY 197 , PS 197 , SA 197 , JO 196 , KW 196 , LB 196 , SD 196 , SY 196 , AE 196 , YE 196 , BH 195 , OM 195 , QA 195 , MA 189 , TN 189 , DZ 188 IQ 200 , JO 200 , KW 200 , BH 200 , OM 200 , PS 200 , QA 200 , SA 200 , AE 200 , YE 200 , LB 199 , SY 199 , LY 197 , SD 197 , EG 196 , MA 190 , TN 190 , DZ 188 Table G1: The number of times a country label is predicted by the model for the 200 sentences of the six anchor dialects of the MADAR CORPUS-26 corpus 23 : Beirut (BEI), Cairo (CAI), Doha (DOH), Rabat (RAB), Tunis (TUN), and MSA. Each dialect’s representative country is in bold , and the country’s same-region countries are underlined , following the regional grouping of 9 . Note #1: the DID-country model is a single-label DI model trained on the MADAR corpus. Its province-level predictions are mapped into country-level ones, with MSA predictions mapped to all the 18 country-level dialects we consider. Note #2: Country abbreviations - AE : UAE, BH : Bahrain, DZ : Algeria, EG : Egypt, IQ : Iraq, JO : Jordan, KW : Kuwait, LB : Lebanon, LY : Libya, MA : Morocco, OM : Oman, PS : Palestine, QA : Qatar, SA : Saudi Arabia, SD : Sudan, SY : Syria, TN : Tunisia, YE : Yemen.