Reasoning Beyond Labels: Measuring LLM Sentiment in Low-Resource, Culturally Nuanced Contexts

Millicent Ochieng, Anja Thieme, Ignatius Ezeani, Risa Ueno, Samuel Maina, Keshet Ronen, Javier González, Jacki O’Neill · 2025

Read Reasoning Beyond Labels: Measuring LLM Sentiment in Low-Resource, Culturally Nuanced Contexts by Millicent Ochieng, Anja Thieme, Ignatius Ezeani, Risa Ueno, Samuel Maina, Keshet Ronen, Javier González, Jacki O’Neill (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-04199.txt.

Introduction

Millicent Ochieng and co-authors probe how well language models read feelings in the code-mixed WhatsApp chats of Nairobi youth health groups, in a paper from 2025. Standard sentiment tools assume fixed labels and universal ways of showing emotion. Real chat in low-resource settings mixes languages and local idiom, breaking those assumptions.

Their diagnostic framework treats sentiment as a cultural construct. They test models on human-annotated messages, on flipped versions with reversed sentiment, and on the quality of the models' own explanations, scored by rubric. Top-tier models reasoned stably; open models often wobbled under ambiguity or sentiment flips. The authors argue for culturally sensitive, reasoning-aware evaluation before such tools judge real communities.

Catalog record

CREATORMillicent Ochieng, Anja Thieme, Ignatius Ezeani, Risa Ueno, Samuel Maina, Keshet Ronen, Javier González, Jacki O’Neill
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-04199

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reasoning Beyond Labels: Measuring LLM Sentiment in Low-Resource, Culturally Nuanced Contexts Millicent Ochieng, Anja Thieme, Ignatius Ezeani, Risa Ueno, Samuel Maina, Keshet Ronen, Javier González, Jacki O’Neill Abstract Sentiment analysis in low-resource, culturally nuanced contexts challenges conventional NLP approaches that assume fixed labels and universal affective expressions. We present a diagnostic framework that treats sentiment as a context-dependent, culturally embedded construct, and evaluate how large language models (LLMs) reason about sentiment in informal, code-mixed WhatsApp messages from Nairobi youth health groups. Using a combination of human-annotated data, sentiment-flipped counterfactuals, and rubric-based explanation evaluation, we probe LLM interpretability, robustness, and alignment with human reasoning. Framing our evaluation through a social-science measurement lens, we operationalize and interrogate LLMs outputs as an instrument for measuring the abstract concept of sentiment. Our findings reveal significant variation in model reasoning quality, with top-tier LLMs demonstrating interpretive stability, while open models often falter under ambiguity or sentiment shifts. This work highlights the need for culturally sensitive, reasoning-aware AI evaluation in complex, real-world communication. 1 Introduction Sentiment analysis is a prevalent NLP technique used to obtain meaningful information and semantics from text Onyenwe et al. 2020 . It is often conflated with emotion detection, which focuses on the psychological state or mood articulated in messages Nandwani and Verma 2021 ; or opinion mining such as consumer sentiment Burnham 2024 . Instead, sentiment analysis primarily determines polarity in the intent behind a written message as positive, negative, or neutral Nandwani and Verma 2021 . Speech Act Theory by Austin 1975 further differentiates between the form of an utterance (locution); its purpose and effect on the hearer (illocution); and the real-world impact (perlocution). This theory highlights that what a reader understands from a message depends on words choices; their individual meanings; ordering; as well as lexical or syntactic variations. Combined, these introduce significant ambiguity on how inferences about a message are drawn Corvi et al. 2025 . Furthermore, the social semiotics theory by Halliday 2014 emphasizes that sentiment is not just a linguistic phenomenon, and also deeply embedded in social and cultural contexts; highlighting how emotions are conveyed and interpreted based on cultural norms and values Zhang 2024 . In this paper, we acknowledge that interpreting or measuring sentiment can be difficult —particularly in informal, multilingual, under-resourced, and culturally nuanced communication contexts. Expressions of emotion and attitude are shaped by local language practices, shared cultural knowledge, and interactional context Matsumoto 1990 ; Lindquist 2021 ; Fang et al. 2022 . In real-world communications such as youth chat, social media, or hyperlocal exchanges among multilingual speakers, language is frequently code-mixed 1 1 1 The practice where multilingual speakers fluidly shift between languages in conversation , fluid, and shaped by the moment—that is, influenced by who is speaking and who is listening, the topic being discussed, the speaker’s emotional tone, or intentions at that time, and the setting (e.g., online chat vs. in-person talk). Meanings are negotiated, implicit, and frequently ambiguous—making sentiment difficult to interpret, even for humans, especially when removed from the original platform or context of exchange O’Neill and Martin 2003 . These complexities do not just complicate classification—they challenge the very measurement of sentiment. As argued by Wallach et al. 2025 , evaluating GenAI models requires treating such tasks as a social science measurement challenge, where abstract, culturally-contested concepts must be systematically defined and carefully connected to observable indicators. Example Complexities Annotator 1 Annotator 2 uyu sasa anachoma manzee “ this guy is now messing up, bro” Shorthand: “uyu” instead of “huyu” Urban slang (Sheng): “anachoma”, “manzee” Tone: friendly teasing, mockery, or social critique Emoji use: Code-mixing: Swahili-Sheng blend Cultural reference: Assumes shared understanding of local slang, social behaviors, and norms Label: Negative Notes: We see the ridicule and embarrassment from the persona and the audience despite the laugh. Label: Positive Notes: Expresses criticism with amusement portrayed with laughing emojis. Nmeacha izea “I’m sorry, I have stopped” Code-mixing: Swahili-Sheng blend Urban slang (Sheng): “izea” Shorthand: “nmeacha” Tone: flat or understated Ambiguity: lacks strong emotional cues Label: Neutral Notes: We see a casual apology that doesn’t express strong emotion. Label: Positive Notes: Speaker is apologetic and remorseful. U can’t see the future but God can “You can’t see the future but God can” Shorthand: “U” for “you” Religious expression: appeals to divine foresight Tone: factual or reassuring Cultural context: common in faith-based communication Ambiguity: sentiment depends on interpretation of tone/intention Label: Neutral Notes: A remark without strong personal emotion. Label: Positive Notes: Speaker expresses trust in God, offering reassurance. Hello, guys yani mko tu na mmenyamaza?? “Hello, guys are online and you are quiet?” Code-mixing: Swahili-English blend Tone: questioning, possibly sarcastic Social cue: expectation of group participation Ambiguity: tone varies between concern and frustration Label: Negative Notes: We see disappointment and negative shock from the persona on why people are so quiet. Label: Neutral Notes: Expresses concern and curiosity on the silence of the group. Yes I eat too much iz it normal “Yes I eat too much is it normal” Shorthand: “iz” for “is”, informal tone Self-disclosure: reveals possible worry Ambiguity: phrased as a question, unclear tone; genuine concern vs casual comment Label: Negative Notes: We see worry and distress about too much eating, suggests a negative sentiment. Label: Neutral Notes: Question seeking clarification. Table 1: Examples of annotator disagreement illustrating cultural and linguistic complexities. In our work, we treat sentiment not as a fixed label, but as a context-dependent expression of intent. It may be explicit (e.g., “I’m so angry right now”), but more often in our dataset, it appears through muted cues (e.g., “You’re always online”)—subtle, culturally and contextually situated, and open to interpretation. We define ambiguity as cases where the intended sentiment is unclear, underspecified, open to multiple readings, or leads to disagreement even among culturally fluent, context-aware annotators—not because the language is misunderstood, but due to differing interpretations of tone or social context (see Table 1 ). We use cultural nuance to describe how language practices, religious or affective expressions, and shared social knowledge shape how sentiment is conveyed and perceived. In our dataset, such nuance is embedded within: practices of code-mixing (e.g., “kama hauko school shindaapo!!”) 2 2 2 Swahili-English: “If you’re not in school, stay there.” While casual, this often conveys a dismissive stance, reflecting cultural norm that link education to intellectual legitimacy. ; local shorthand (e.g., mm for mimi) 3 3 3 mimi means “me” in Swahili ; emoji-only or emoji-enhanced messages via graphical symbols (e.g.,“ ”) or their textual counterpart the emoticon (e.g., ":)") Liu et al. 2021 ; Yoo and Rayz 2021 ; irony ; and youth-specific slang (Sheng) 4 4 4 An urban slang spoken by youth in Kenya, blending Swahili, English, and local languages. . These elements are often combined to produce rich, but difficult-to-classify, sentiment signals; and these cultural complexities are evident throughout our dataset (see Table 1 and Appendix Table 11 ) So far, standard sentiment analysis treats sentiment as a fixed classification problem, assuming a single, context-independent “ground truth” Mohammad 2017 ; Wankhade et al. 2022 ; Sharma et al. 2024 . While LLMs have transformed NLP Brown et al. 2020 ; Touvron et al. 2023 ; Jiang et al. 2023 ; OpenAI 2023

Read the complete text