Polite on the Surface, Wrong in Practice: A Curated Dataset for Fixing Honorific Failures in Multilingual Bangla Generation

Md. Asaduzzaman Shuvo, Mahedi Hasan · 2026

Read Polite on the Surface, Wrong in Practice: A Curated Dataset for Fixing Honorific Failures in Multilingual Bangla Generation by Md. Asaduzzaman Shuvo, Mahedi Hasan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22487.txt.

Introduction

Shuvo and team introduce BLADE, a culturally aligned dataset for Bangla app and talk generation. Current models struggle with structural shifts, regional idioms and honorific match in low-resource Bangla. The dataset holds 4,196 pairs, with a benchmark set. Leading open models, including DeepSeek-8B and LLaMA-3.2-3B, are tuned with LoRA adapters in a 4-bit NormalFloat setup. Tuned models improve a lot in structural fidelity and honorific alignment, offering a strong benchmark for low-resource multilingual generation.

Catalog record

CREATORMd. Asaduzzaman Shuvo, Mahedi Hasan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49184
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.22487

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Polite on the Surface, Wrong in Practice: A Curated Dataset for Fixing Honorific Failures in Multilingual Bangla Generation Md. Asaduzzaman Shuvo, Mahedi Hasan Abstract Recent advances in Multilingual Large Language Models (MLLMs) have significantly enhanced cross-lingual conversational capabilities, yet modeling culturally nuanced and context-dependent communication remains a critical bottleneck. Specifically, existing state-of-the-art models exhibit a severe pragmatic gap when handling structural variations, regional idioms, and honorific consistencies in low-resource contexts like Bangla. To address this limitation, we introduce a novel, culturally aligned instruction-tuning dataset for BangLa Application and DialoguE generation - BLADE and benchmarking framework comprising 4,196 4,196 meticulously curated interaction pairs. We leverage this resource to systematically fine-tune and evaluate leading open-weight architectures, including DeepSeek-8B and LLaMA-3.2-3B, utilizing parameter-efficient fine-tuning via LoRA adapters in a 4-bit NormalFloat (NF4) quantization framework. Our empirical evaluations demonstrate that models fine-tuned on our dataset yield substantial improvements in structural fidelity and honorific alignment, providing a rigorous benchmark for bridging pragmatic disparities in low-resource multilingual text generation. Code and dataset: https://github.com/ashuvo25/Bangla_Application_LLM/tree/main [ BoldFont = TeXGyreTermesX-Bold.otf, ItalicFont = TeXGyreTermesX-Italic.otf, BoldItalicFont = TeXGyreTermesX-BoldItalic.otf ] 1 Introduction Large Language Models (LLMs) have fundamentally transformed multilingual text generation ( 1 ; 24 ) , yet for low-resource languages this progress remains superficial. The multilingual curse manifests not as vocabulary gaps, but as a deeper pragmatic gap , the divide between surface fluency and functional usability a structural limitation affecting any language whose institutional writing encodes social relationships grammatically. Our findings extend beyond Bangla, the failure of general-purpose multilingual pretraining to internalize register-aware, format-sensitive conventions is a structural limitation affecting any language whose institutional writing encodes social relationships grammatically. Bangla, the fifth most spoken language worldwide with over 230 million native speakers, makes this failure concrete. Current multilingual models produce Bangla text that appears fluent by surface-level metrics while simultaneously failing every practical test of usability: misplaced document structures, inconsistent honorific registers, and culturally incongruent discourse markers that make outputs institutionally unacceptable ( 15 ; 23 ) . Zero-Shot (Flawed) BLADE-SFT (Correct) মহোদয়, আমি আপনার বিদ্যালয়ের একজন ছাত্র। তুমি কি আমাকে তিন দিনের ছুটি দিতে পারবে? আপনার কাছে বিনীত আবেদন করছি। (English: "Sir, I am a student… Can you [informal] give me leave? I appeal to you [formal] .") মহোদয়, সম্মানপূর্বক বিনীত নিবেদন এই যে, আমি আপনার বিদ্যালয়ের নবম শ্রেণীর একজন নিয়মিত ছাত্র। আগামী তিন দিনের জন্য আপনার নিকট অগ্রিম ছুটির আবেদন করছি। (English: "Sir… I hereby appeal to you [consistent formal] for three days of advance leave.") Table 1: Comparison of generated text from a zero-shot baseline and the DeepSeek-8B BLADE-SFT model. While the zero-shot baseline fails to maintain social context by mixing informal (Tumi) and formal (Apnar) pronouns in a professional setting, the fine-tuned model consistently applies the correct formal honorifics required for a student-to-principal application. The root cause is not vocabulary coverage it is the absence of register-aware supervision . Bangla encodes social relationships directly into its grammar through a two-tier honorific system. The formal second-person pronoun আপনি (Apni : You) and its associated verb forms are obligatory in professional, institutional, and elder-addressed writing. The informal counterpart তুমি (Tumi: You) is reserved for peers, close friends, and juniors. Mixing these registers within a single document is not merely stylistically awkward it signals a fundamental breakdown in social competence and renders the document unusable in any formal context ( 23 ) . Table 1 illustrates this failure concretely. A zero-shot baseline model, given the prompt "Write a formal leave application to your school principal" , produces output that oscillates between Apni: you and Tumi: you registers within the same paragraph an error no native speaker would make and no institution would accept. Beyond honorific mismatch, zero-shot models exhibit recurring structural failures: omitting mandatory document components date, subject line, formal salutation, misplacing closings, and using discourse markers appropriate for casual speech in formal writing contexts. These are not edge cases our empirical evaluation shows they occur systematically across all tested architectures. Even the strongest zero-shot baseline, Gemini-2.5-Flash, achieves only 7.88 BLEU, confirming that general-purpose pretraining fails to internalize the format-sensitive and register-aware conventions required by native speakers. To address these limitations, we introduce BLADE (BangLa Applications and DialoguEs) , a specialized resource serving a dual role: (i) an instruction-tuning dataset of 4,196 high-quality, expert-annotated prompt-response pairs spanning 2,008 unique topics, and (ii) a rigorous evaluative benchmark for measuring model adherence to Bangla structural and cultural norms. BLADE was constructed through a principled three-tier acquisition pipeline: government-approved textbooks establish the gold standard for canonical formatting; verified web portals provide real-world usage diversity; and author-synthesized examples, cross-validated by native linguistic experts, address complex honorific alignment scenarios underrepresented in static sources. Our contributions are as follows: • We introduce BLADE , a publicly available instruction-tuning dataset of 4,196 expert-annotated pairs spanning 2,008 unique topics, built through a three-tier pipeline with native linguistic expert validation, covering educational, professional, and conversational Bangla generation. • We benchmark the pragmatic gap in five state-of-the-art multilingual LLMs under zero-shot conditions, systematically documenting pragmatic register failures and document structure violations across all tested architectures. • We demonstrate that targeted SFT on BLADE yields consistent gains across automatic metrics (BLEU 17.73; chrF 46+), human expert evaluation ( > 4.6/5 on all dimensions), and LLM-as-judge scoring (8.30/10), establishing that cultural data specificity outweighs model scale for low-resource pragmatic generation. 2 Related Work Multilingual LLMs and the Low-Resource Gap. Progress in LLMs has been driven by web-scale corpora and instruction datasets such as Common Crawl, Alpaca , and Dolly , with multilingual coverage expanded by projects like HPLT ( 7 ) . Despite these gains, a well-documented 25–35% instruction-following accuracy gap persists between high- and low-resource languages ( 25 ) , and LLMs consistently fail to outperform classical baselines on extremely low-resource languages in zero-shot settings ( 6 ) . Critically, machine-translated instruction data systematically underestimates model capability compared to natively localized equivalents ( 4 ) , motivating our choice to construct BLADE from human-curated, institutionally grounded Bangla sources. Honorific Systems and Pragmatic Competence. A deeper multilingual failure concerns pragmatic competence: generating text that is socially and institutionally appropriate, not merely grammatical. For languages that grammaticalize social relationships into morphology, register consistency is a necessary condition for functional usability. Japanese LLMs fail to generalize honorific patterns across novel syntactic structures ( 22 ) , and even frontier models including GPT-4o underperform on high-politeness Javanese registers underrepresented in web-scale data ( 8 ) . Arabic evaluation reveals the same failure: surface fluency does not imply cultural coherence ( 3 ) . Topic Content (Bangla) English Translation অসুস্থতার কারণে ছুটির আবেদন (Application for sick leave) তারিখঃ ১৮/১০/২০২৪ খ্রিঃ বরাবর প্রধান শিক্ষক ছোলমাইদ হাই স্কুল এন্ড কলেজ ভাটারা, ঢাকা-১২১২ বিষয়ঃ বিদ্যালয়ে অনুপস্থিতির জন্য প্রধান শিক্ষকের নিকট আবেদন। স্যার …… Date: 18/10/2024 AD To Headmaster Solmaid High School and College Vatara, Dhaka-1212 Subject: Application to the Headmaster regarding absence from school. Sir…… Table 2: Example BLADE entry with English translation, illustrating canonical document structure. These findings establish that Bangla’s honorific mismatch problem, documented in Table 1 , is a cross-linguistically shared structural failure, not a Bangla-specific anomaly. Bangla NLP Resources. Bangla NLP has progressed from encoder models such as BanglaBERT ( 5 ) to native generative models including TigerLLM ( 21 ) and TituLLMs

Read the complete text