Deciphering Emotions in Children Storybooks: A Comparative Analysis of Multimodal LLMs in Educational Applications

Bushra Asseri, Estabraq Abdelaziz, Maha Al Mogren, Tayef Alhefdhi, Areej Al-Wabil · 2025

Read Deciphering Emotions in Children Storybooks: A Comparative Analysis of Multimodal LLMs in Educational Applications by Bushra Asseri, Estabraq Abdelaziz, Maha Al Mogren, Tayef Alhefdhi, Areej Al-Wabil (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-18201.txt.

Introduction

Bushra Asseri and four co-authors test AI reading of picture books in a 2025 study. They used Arabic storybooks meant for young children. Two large multimodal models judged the feelings shown in the pictures. The team compared their calls with labels made by human readers.

Seventy five images came from seven storybooks. Each model tried three prompting styles, from no examples to step by step reasoning. One model scored far higher than the other across all styles. Its best score came with the step by step prompts. Most errors flipped a feeling to its opposite in tone. Both models stumbled on culture specific feelings and unclear scenes. The authors call for training that respects local culture in school tools.

Catalog record

CREATORBushra Asseri, Estabraq Abdelaziz, Maha Al Mogren, Tayef Alhefdhi, Areej Al-Wabil
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-18201

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Deciphering Emotions in Children Storybooks: A Comparative Analysis of Multimodal LLMs in Educational Applications Bushra Asseri, Estabraq Abdelaziz, Maha Al Mogren, Tayef Alhefdhi, Areej Al-Wabil Abstract Emotion recognition capabilities in multimodal AI systems are crucial for developing culturally responsive educational technologies, yet remain underexplored for Arabic language contexts where culturally appropriate learning tools are critically needed. This study evaluates the emotion recognition performance of two advanced multimodal large language models, GPT-4o and Gemini 1.5 Pro, when processing Arabic children’s storybook illustrations. We assessed both models across three prompting strategies (zero-shot, few-shot, and chain-of-thought) using 75 images from seven Arabic storybooks, comparing model predictions with human annotations based on Plutchik’s emotional framework. GPT-4o consistently outperformed Gemini across all conditions, achieving the highest macro F1-score of 59% with chain-of-thought prompting compared to Gemini’s best performance of 43%. Error analysis revealed systematic misclassification patterns, with valence inversions accounting for 60.7% of errors, while both models struggled with culturally nuanced emotions and ambiguous narrative contexts. These findings highlight fundamental limitations in current models’ cultural understanding and emphasize the need for culturally sensitive training approaches to develop effective emotion-aware educational technologies for Arabic-speaking learners. \setcode utf8 K eywords Multimodal Large Language Models (MLLMs) \cdot Digital Storybooks \cdot Educational Technology \cdot Culturally Responsive Education \cdot Prompting Techniques \cdot Emotional Intelligence 1 Introduction Understanding emotions through visual forms is essential for effective communication and learning, especially in early childhood ( denham_early_2012 ; pons_emotion_2004 ) . As the role of Artificial Intelligence (AI) in education continues to grow, the ability to interpret emotions in illustrated children’s books becomes increasingly important for developing culturally responsive educational technologies ( chen_application_2020 ) . Recent advances in MLLMs, such as GPT-4o and Gemini, have significantly improved visual-textual reasoning capabilities ( alayrac_flamingo_2022 ) , yet their application to non-English educational contexts remains limited ( joshi_state_2020 ) . Arabic children’s literature presents unique challenges for computational emotion recognition due to its expressive language, cultural symbolism, and artistic style ( habash_introduction_2010 ) . Despite the educational importance of emotion recognition in early literacy development highlighted by denham_plays_2010 , there exists a significant gap in our understanding of how well current AI systems can interpret emotional content in Arabic visual narratives. This study addresses this research gap by systematically evaluating the emotion recognition capabilities of two advanced MLLMs, GPT-4o and Gemini Pro 1.5, when processing illustrations from Arabic children’s storybooks. The primary aim of this study is to determine the accuracy and effectiveness of Large Language Models in identifying emotions in images from Arabic children’s storybooks. To accomplish this aim, we pursue several specific objectives: to evaluate and compare the effectiveness of different prompting strategies (zero-shot, few-shot, chain-of-thought) on MLLMs’ emotion recognition accuracy in Arabic visual narratives; to identify and categorize common error patterns when MLLMs interpret emotional content in Arabic children’s literature; and to assess MLLMs’ performance against human consensus when interpreting ambiguous or culturally nuanced emotional expressions. To achieve these objectives, we compare the models’ performance across three distinct prompting strategies using a dataset of 75 images from seven Arabic storybooks, with human-annotated emotions based on Plutchik’s framework serving as ground truth ( plutchik_nature_2001 ) . Beyond simple accuracy metrics, we analyze cases where models struggle with narrative complexity or misinterpret subtle emotional expressions, providing insights into the challenges of cross-cultural emotion recognition in educational AI ( wei_chain–thought_2022 ) . This research makes several contributions to the fields of educational technology and Arabic digital literacy: (1) it presents the first systematic evaluation of MLLMs for emotion recognition in Arabic children’s literature; (2) it identifies specific patterns of success and failure in cross-cultural emotion recognition tasks; and (3) it provides actionable recommendations for developing more culturally sensitive AI systems for Arabic literacy education ( bender_data_2018 ) . Through exploratory experiments expanding the emotional label space and isolating individual characters from their narrative contexts, we offer additional insights into the models’ emotional reasoning capabilities. The findings of this study have significant implications for the design of emotion-aware educational technologies that can effectively support Arabic literacy acquisition and social-emotional learning in culturally appropriate ways. 2 Literature Review 2.1 Emotion Recognition in Educational Technologies Affective computing has facilitated the integration of emotion recognition within educational technology, enabling adaptive systems to personalize interactions by detecting engagement, confusion, frustration, and joy. Research shows that intelligent tutoring systems with emotion recognition capabilities increase student engagement and retention through responsive feedback. Recent studies highlight multimodal approaches that combine facial expressions, eye movements, and biosignal data, with empirical evidence showing significant improvements in learning outcomes when integrated into educational environments ( al-laith_monitoring_2021 ) . These technological applications are grounded in established educational frameworks. Vygotsky’s concept of the Zone of Proximal Development aligns with how adaptive systems adjust learning tasks based on detected emotional states ( vygotsky_mind_1978 ) , while Piaget’s theory of cognitive development intersects with emotional development through the influence of emotional states on attention and focus ( piaget_intelligence_1981 ; denham_plays_2010 ) . These theoretical connections are especially relevant in cross-cultural contexts such as Arabic educational settings, where sociocultural factors significantly influence emotional expression and interpretation ( habash_introduction_2010 ; matsumoto_evidence_2011 ) . 2.2 Arabic Children’s Literature and Emotional Content Arabic children’s literature presents unique challenges for computational emotion recognition due to its expressive language, cultural symbolism, and implicit emotional cues. The poetic tradition in Arabic literature contributes to its emotional complexity through specific formal structures and elaborate metaphors, with distinctive symbolism: desert imagery representing resilience, nature elements mirroring emotional states, and love metaphors expressing profound longing. This rich symbolic language creates challenges for computational systems trained primarily on literal expressions. Additional challenges include the morphological complexity and diglossic nature of Arabic, where Modern Standard Arabic coexists with more than 25 regional dialects that differ morphologically, phonologically, and lexically. Children’s literature often contains elements of both formal and dialectal forms, particularly in dialogue, creating a complex linguistic landscape for emotion detection ( alhuzali_spanemo_2021 ) . Visual elements often convey emotional subtleties within Arabic children’s stories, highlighting the necessity for AI models capable of nuanced multimodal understanding. Recent computational approaches have developed specialized Arabic emotion lexicons for basic emotions, demonstrating the need for culturally adapted resources ( abdul-mageed_emonet_2017 ) . Preliminary studies suggest that existing LLMs sometimes misinterpret culturally contextual imagery, underscoring the importance of culturally anchored datasets and annotations ( abdul-mageed_arbert_2021 ; alhuzali_spanemo_2021 ) . 2.3 Plutchik’s Emotion Model in Cross-Cultural Contexts Plutchik’s Wheel of Emotions, comprising eight fundamental emotions (joy, trust, fear, surprise, sadness, disgust, anger, anticipation), serves as our analytical framework ( plutchik_nature_2001 ; mohammad_crowdsourcing_2013 ) . Each emotion is represented at three intensity levels, with opposites arranged diametrically and potential combinations identified (e.g., joy + trust = love) ( plutchik_nature_2001 ) . However, the model’s presumed universality requires scrutiny given cultural variability in emotional expression. For instance, emotions like ”trust” ( \< ثقة¿) within Arabic narratives encompass broader concepts of reassurance, confidence, and relational sensibilities, diverging from Plutchik’s Western-centric definitions ( al-laith_monitoring_2021 ) . When applied to Arabic emotion detection, researchers have adapted the framework by developing specialized seed lexicons corresponding to the eight primary emotions ( al-laith_monitoring_2021 ; nfaoui_evaluating_2024 ) . Cross-cultural studies provide further evidence of both universality and cultural specificity in emotion recognition. chronaki_development_2018 demonstrated that while

Read the complete text