CAPITU: A Benchmark for Evaluating Instruction-Following in Brazilian Portuguese with Literary Context

Giovana Kerche Bonás, Roseval Malaquias Junior, Marcos Piau, Thiago Laitz, Thales Sales Almeida, Hugo Abonizio, Celio Larcher, Ramon Pires, Rodrigo Nogueira · 2026

Read CAPITU: A Benchmark for Evaluating Instruction-Following in Brazilian Portuguese with Literary Context by Giovana Kerche Bonás, Roseval Malaquias Junior, Marcos Piau, Thiago Laitz, Thales Sales Almeida, Hugo Abonizio, Celio Larcher, Ramon Pires, Rodrigo Nogueira (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22576.txt.

Introduction

Bonás and co-authors introduce CAPITU, a benchmark for instruction-following in Brazilian Portuguese. Every task sits inside one of eight classic Brazilian novels, pairing checkable rules with cultural content. The set holds 59 instruction types in seven groups, all scored automatically with no human or model judges. Some rules test Portuguese word patterns that English benchmarks never touch. Eighteen leading models were tested in single-turn and multi-turn chats. Frontier reasoning models scored near perfect, while cheaper Portuguese-tuned models stayed competitive.

Catalog record

CREATORGiovana Kerche Bonás, Roseval Malaquias Junior, Marcos Piau, Thiago Laitz, Thales Sales Almeida, Hugo Abonizio, Celio Larcher, Ramon Pires, Rodrigo Nogueira
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73288
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.22576

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CAPITU: A Benchmark for Evaluating Instruction-Following in Brazilian Portuguese with Literary Context Giovana Kerche Bonás, Roseval Malaquias Junior, Marcos Piau, Thiago Laitz, Thales Sales Almeida, Hugo Abonizio, Celio Larcher, Ramon Pires, Rodrigo Nogueira Abstract We introduce CAPITU, a benchmark for evaluating instruction-following capabilities of Large Language Models (LLMs) in Brazilian Portuguese. Unlike existing benchmarks that focus on English or use generic prompts, CAPITU contextualizes all tasks within eight canonical works of Brazilian literature, combining verifiable instruction constraints with culturally-grounded content. The benchmark comprises 59 instruction types organized into seven categories, all designed to be automatically verifiable without requiring LLM judges or human evaluation. Instruction types include Portuguese-specific linguistic constraints (word termination patterns like -ando/-endo/-indo, -inho/-inha, -mente) and structural requirements. We evaluate 18 state-of-the-art models across single-turn and multi-turn settings. Our results show that frontier reasoning models achieve strong performance (GPT-5.2 with reasoning: 98.5% strict accuracy), while Portuguese-specialized models offer competitive cost-efficiency (Sabiazinho-4: 87.0% at $0.13 vs Claude-Haiku-4.5: 73.5% at $1.12). Multi-turn evaluation reveals significant variation in constraint persistence, with conversation-level accuracy ranging from 60% to 96% across models. We identify specific challenges in morphological constraints, exact counting, and constraint persistence degradation across turns. We release the complete benchmark, evaluation code, and baseline results to facilitate research on instruction-following in Portuguese. 1 Introduction Large Language Models (LLMs) exhibit versatility across a broad spectrum of tasks, from creative writing to complex problem-solving. However, their practical deployment hinges on their ability to follow precise instructions without compromising output quality. Evaluating this capability through explicit constraints offers a quantifiable measure of model controllability. The capacity to satisfy rigid structural requirements validates whether a model can preserve coherence and utility under strict boundary conditions, thereby indicating greater robustness [ 11 ] . The importance of instruction-following evaluation has driven the development of specialized benchmarks. IFEval [ 18 ] pioneered a paradigm of verifiable instructions with 25 constraint types that can be automatically checked without subjective judgment, establishing a foundation for systematic evaluation in English. Subsequent work has expanded this paradigm: IFBench [ 13 ] introduced 58 new instruction types to test generalization to novel constraints, FollowBench [ 7 ] organized complexity into hierarchical levels, and Multi-IF [ 6 ] extended this approach to eight languages, including Portuguese, through the translation of English prompts. Despite these advances, instruction-following evaluation for Portuguese remains comparatively underexplored. While Multi-IF [ 6 ] provides a baseline for multilingual coverage, evaluation sets built primarily via direct translation may not fully reflect language-specific phenomena or culturally grounded contexts. Portuguese exhibits rich morphological features and discourse patterns that differ fundamentally from English, necessitating a language-specific instruction design rather than adaptation. The limitations of translation-based evaluation become particularly apparent for Portuguese. First, linguistic specificity: Portuguese exhibits morphological and syntactic features that cannot be adequately assessed through translated English instructions—its productive diminutive system (-inho/-zinho), gerund constructions, and adverb formation (-mente) require native instruction design [ 5 ] . Second, discourse conventions: translated instructions reflect English writing patterns and rhetorical norms, while Portuguese has distinct conventions for formality, argumentation, and connective usage. Third, evaluation authenticity: translated instructions may introduce artifacts or miss language-specific constraints central to Portuguese writing. We address these challenges by introducing CAPITU ( Contextual Assessment of Portuguese Instruction-following Through Literary Understanding ), an instruction-following benchmark designed natively for Brazilian Portuguese. Rather than translating existing English benchmarks, CAPITU contextualizes its prompts within eight canonical works of Brazilian literature spanning Romanticism, Realism, Naturalism, Regionalism and Modernism. This literary framing provides natural, coherent scenarios for generation tasks, while evaluation focuses on verifiable constraint satisfaction. CAPITU uses 59 instruction types, including Portuguese-specific constraints targeting morphological phenomena such as diminutive suffixes and gerund forms that lack direct English equivalents. CAPITU is available at https://github.com/maritaca-ai/capitu . A key design principle of CAPITU is automatic verification: all 59 instruction types are automatically verifiable using string operations, regular expressions, and lexicon matching. This eliminates the need for LLM judges or human annotators to evaluate instruction compliance, ensuring reproducible and objective evaluation. While LLM-as-judge approaches have shown strong correlation with human evaluation [ 17 , 10 ] , they introduce non-determinism and cost overhead. CAPITU’s literary grounding serves multiple purposes. Brazilian and Portuguese literary classics provide naturally occurring text with varied narrative techniques, regional varieties, and historical registers that contextualize instruction-following within linguistically rich material. This paper is organized as follows: Section 2 reviews related work in instruction-following evaluation, multilingual benchmarks, and literary Natural Language Processing (NLP). Section 3 describes CAPITU’s design principles, instruction taxonomy, and evaluation framework. Section 4 provides benchmark statistics. Section 5 presents our experimental setup and baseline results across multiple models. Section 6 analyzes performance patterns and failure modes. Section 7 discusses implications, limitations, and future directions. We conclude in Section 8 . 1.1 Contributions Figure 1 summarizes our four main contributions: (1) a Portuguese-specific instruction taxonomy with 59 automatically verifiable types across 7 categories, including morphological patterns unique to Portuguese; (2) culturally-contextualized prompts grounded in 8 canonical works of Brazilian literature; (3) a multi-dimensional assessment framework with difficulty levels, multi-turn evaluation, and optional coherence scoring; and (4) a comprehensive evaluation suite with 500 prompts: 200 single-turn prompts and 100 multi-turn dialogues with three turns each (300 turns total), baselines for 18 models, and open-source code. Figure 1 : Overview of CAPITU’s main contributions. 2 Related Work 2.1 Instruction-Following Evaluation The systematic evaluation of instruction-following capabilities emerged as LLMs became increasingly deployed in applications requiring precise output control. 18 introduced IFEval, establishing a paradigm of verifiable instructions with constraints that can be automatically checked without subjective judgment. Their 25 instruction types cover word counts, formatting requirements, keyword inclusion, and structural constraints, evaluated on 541 English prompts. Building on this foundation, 7 proposed FollowBench, which uses a multi-level mechanism to incrementally add constraints, creating five levels (1–5 constraints) across five constraint categories, evaluated in single-round settings over 820 curated instructions. 14 contributed InFoBench with 500 prompts, focusing on decomposed instruction evaluation to assess granular capabilities and identify specific failure modes in instruction-following. 13 extended this paradigm with IFBench, introducing 58 new verifiable out-of-domain constraint templates paired with 300 prompts to test generalization to unseen constraint formulations. They show models often overfit prior constraint sets and struggle on novel constraints. Multi-IF [ 6 ] addressed the multilingual dimension by translating IFEval’s 25 instruction types to seven languages (Chinese, French, Hindi, Italian, Portuguese, Russian, and Spanish) plus English, creating a dataset of 3,192 prompts. The benchmark extends evaluation to multi-turn conversations with three turns, testing whether models maintain instruction adherence across dialogue. While Multi-IF represents an important milestone in multilingual instruction-following evaluation, its translation-based approach has inherent limitations: the 25 instruction types are designed for English and may not capture language-specific phenomena, prompts retain English-centric contexts and cultural references, and the evaluation cannot assess linguistic features unique to target languages. For Portuguese specifically, Multi-IF does not evaluate constraints related to word termination patterns, Portuguese discourse markers, or culturally grounded content, gaps that CAPITU addresses through native Portuguese instruction design. 2.2 Native and Culturally-Grounded Benchmarks The NLP community has begun to shift focus from broad multilingual understanding to the evaluation of native instruction-following capabilities, recognizing that translation-based constraints often fail to capture the complexity of specific languages. Recent benchmarks have emerged to

Read the complete text