Structured Prompting for Arabic Essay Proficiency: A Trait-Centric Evaluation Approach

2026

Read Structured Prompting for Arabic Essay Proficiency: A Trait-Centric Evaluation Approach (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19668.txt.

Introduction

Al Mandhari and co-authors present a structured prompting framework for scoring Arabic essays by trait. Tools for Arabic essay scoring are scarce and hard to scale. Their three-tier strategy covers standard, hybrid, and rubric-guided prompts. The hybrid tier mimics a panel of trait specialist raters. The rubric tier adds scored exemplars to align the model. Eight models were tested on the QAES data set in zero-shot and few-shot modes. Fanar-1-9B-Instruct agreed best with human scores. Rubric-guided prompts helped every trait and model.

Catalog record

YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29819
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.19668

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Structured Prompting for Arabic Essay Proficiency: A Trait-Centric Evaluation Approach Abstract This paper presents a novel prompt engineering framework for trait specific Automatic Essay Scoring (AES) in Arabic, leveraging large language models (LLMs) under zero-shot and few-shot configurations. Addressing the scarcity of scalable, linguistically informed AES tools for Arabic, we introduce a three-tier prompting strategy (standard, hybrid, and rubric-guided) that guides LLMs in evaluating distinct language proficiency traits such as organization, vocabulary, development, and style. The hybrid approach simulates multi-agent evaluation with trait specialist raters, while the rubric-guided method incorporates scored exemplars to enhance model alignment. In zero and few-shot settings, we evaluate eight LLMs on the QAES dataset, the first publicly available Arabic AES resource with trait level annotations. Experimental results using Quadratic Weighted Kappa (QWK) and Confidence Intervals show that Fanar-1-9B-Instruct achieves the highest trait level agreement in both zero and few-shot prompting (QWK = 0.28 and CI = 0.41), with rubric-guided prompting yielding consistent gains across all traits and models. Discourse-level traits such as Development and Style showed the greatest improvements. These findings confirm that structured prompting, not model scale alone, enables effective AES in Arabic. Our study presents the first comprehensive framework for proficiency oriented Arabic AES and sets the foundation for scalable assessment in low resource educational contexts. Keywords: Arabic AES, Prompt engineering, Trait evaluation Structured Prompting for Arabic Essay Proficiency: A Trait-Centric Evaluation Approach Salim Al Mandhari 1 , Hieu Pham Dinh 2 , Mo El-Haj {}^{1},^{2} , Paul Rayson {}^{1},^{2} 1 Lancaster University, UK 2 VinUniversity, Vietnam s.m.almandhari, m.el-haj, [email protected] 24hieu.pd, elhaj.m, [email protected] Abstract content 1. Introduction AES has emerged as a critical tool in educational assessment, enabling scalable and consistent evaluation of language performance. Although substantial progress has been made in AES for high-resource languages such as English, development for Arabic remains limited by the scarcity of annotated datasets, the linguistic complexity of Arabic, and the lack of tools assessing proficiency based on distinct linguistic skills. Existing Arabic AES systems predominantly focus on holistic scoring or content correctness, often neglecting nuanced aspects of proficiency such as organization, vocabulary richness, idea development, and stylistic appropriateness 11 ; 5 . 13 , for example, demonstrated the use of AraBART for Arabic AES, achieving high accuracy in content-based scoring. However, their focus was limited to correctness of responses, with no evaluation of traits or linguistic proficiency. Conversely, English AES research has advanced toward trait scoring and prompt-based evaluation using LLMs ( 15 ; 8 ) , yet these innovations have not been fully extended to Arabic due to data and methodological constraints. To address this gap, our research investigates how prompt engineering techniques including zero-shot and few-shot configurations can be applied to LLMs for trait-specific scoring of Arabic student essays. Rather than relying on fine-tuning or training custom models, we propose a flexible, rubric-informed prompting strategy designed to work directly with existing LLMs 1 1 1 Code and prompts are available at https://github.com/dinhieufam/Arabic_AES/tree/master . Our contributions are as follows: • Our study is the first to evaluate trait-specific prompt engineering strategies on the QAES dataset, using a linguistically aligned hybrid evaluation framework. • We introduce a novel three-tier prompting framework for AES in Arabic, composed of: (1) Standard prompts for holistic evaluation, (2) Hybrid trait-based prompts, wherein specialist raters assess specific linguistic features (e.g., organization, vocabulary, tone, development), and (3) Rubric-informed prompts that guide the model using explicit scoring criteria for each trait. • We implement few-shot learning for trait-level prompting, providing example-scored responses to calibrate the model’s judgment. This is, to our knowledge, the first application of few-shot prompting for trait-specific scoring in Arabic. • We evaluate our framework on the QAES dataset, a trait-annotated Arabic AES corpus, and demonstrate competitive alignment with human scores across multiple traits in zero and few-shot settings. • We provide qualitative insights into LLM performance variability across traits and prompt types, revealing which linguistic features are most robustly evaluated by current models. This work demonstrates the feasibility of prompt-based scoring as a scalable and adaptable alternative to traditional supervised AES models, particularly in low-resource settings. It also provides a foundation for the future development of LLM-driven educational tools tailored to the Arabic language. 2. Related Work AES has evolved from rule-based and statistical models to transformer-based language models and advanced prompt engineering. This section categorizes prior work by modeling strategies, dataset development, and prompt-based innovations, emphasizing efforts in Arabic AES. Early AES systems for Arabic utilized string and vector similarity methods. For instance, 1 developed a Vector Space Model (VSM) to compare 30 essays with reference texts, which was later extended with Latent Semantic Indexing 2 . While foundational, these approaches lacked granularity and linguistic depth. 22 expanded AES with algorithms such as N-gram and DISCO, achieving moderate accuracy (up to 82%) on short Arabic responses. 16 further enhanced scoring reliability using ensemble models including Random Forest and Adaboost. Transformer models introduced contextual depth to AES. 13 demonstrated this by leveraging AraBART for Arabic AES, achieving 0.88 overall QWK score after training the model across domains using a dataset of over 2,000 essays. Their focus, however, was on evaluating answer correctness in subject-specific contexts rather than linguistic proficiency. Our work, in contrast, centers on assessing language skills through multi-trait analysis. Sentence-level semantic modeling has been effectively explored by 19 , who employed sentence-BERT embeddings with Bi-LSTM and CNNs to capture coherence and cohesion in English AES. The highest QWK score achieved was 0.75. 21 combined psychometric modeling (IRT and CAT) with linear ML to adaptively assess language ability in the Duolingo exam framework. Prompt-based AES has gained traction as a fine-tuning-free method. 15 introduced a zero-shot prompting framework called Multi-Trait Specialization (MTS), enabling LLMs like ChatGPT and LLaMA to achieve competitive trait-level scoring of the TOFEL and ASAP English datasets, gaining QWK scores of 0.43 and 0.35 respectively. The TRATES framework proposed by 12 translated rubrics into guiding questions, improving the scoring alignment of traits using Starling and Gemma models to the average QWK score of 0.59 on the ASAP and ELLIPSE English datasets. 17 tested a four-tier prompt design on English AES using ChatGPT and LLaMA but found these underperformed compared to fine-tuned models, achieving QWK score around 0.57. They did not experiment with few-shot prompting or linguistic trait specialization. 8 implemented feature-based (LR), holistic SOTA (NPCR), single-task learning (STL) and multi-task learning (MTL) models for both holistic and trait scoring using AraBERT on the QAES dataset. The highest QWK score was 0.26 achieved by LR, on the holistic scoring,. This finding can be considered as a baseline for our study. Although no work has been found in literature on prompting design for Arabic proficiency scoring, our approach builds on these insights by proposing a three-tier prompting framework: standard, hybrid, and rubric-based for Arabic AES. We incorporate few-shot examples and trait-decomposed scoring, thereby addressing both scoring fidelity and pedagogical interpretability without model fine-tuning. The Arabic AES domain has been limited by resource scarcity. QCAW, introduced by 26 , provides a bilingual corpus of argumentative essays. QAES, built on QCAW, is the first publicly available Arabic dataset annotated by traits such as organization, vocabulary, and grammar 8 . Trait rubrics were derived from official scoring guidelines by the

Read the complete text