WRAVAL – WRiting Assist eVALuation

Gabriel Benedict, Matthew Butler, Naved Merchant, Eetu Salama-Laine · 2026

Read WRAVAL – WRiting Assist eVALuation by Gabriel Benedict, Matthew Butler, Naved Merchant, Eetu Salama-Laine (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03268.txt.

Introduction

Gabriel Benedict and three co-authors argue that standard benchmarks misjudge small language models, and offer a better yardstick. Evaluation has shifted toward reasoning and problem-solving tests, where models under ten billion parameters score three to four times lower than large ones. Yet small models shine at everyday industrial jobs, such as rewriting text in a funnier, more serious, or more professional tone, which those tests never measure.

Their framework, WRAVAL, is built for non-reasoning tasks where no ready-made evaluation dataset exists. It combines new approaches to data generation, prompt tuning, and model-based evaluation to show what task-specific fine-tuning can achieve. The tools target practitioners deploying models in edge and private computing settings, where small models are often the only option.

Catalog record

CREATORGabriel Benedict, Matthew Butler, Naved Merchant, Eetu Salama-Laine
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-03268

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

WRAVAL – WRiting Assist eVALuation Gabriel Benedict, Matthew Butler, Naved Merchant, Eetu Salama-Laine Abstract The emergence of Large Language Models (LLMs) has shifted language model evaluation toward reasoning and problem-solving tasks as measures of general intelligence. Small Language Models (SLMs) – defined here as models under 10B parameters – typically score 3-4 times lower than LLMs on these metrics. However, we demonstrate that these evaluations fail to capture SLMs’ effectiveness in common industrial applications, such as tone modification tasks (e.g., funny, serious, professional). We propose an evaluation framework specifically designed to highlight SLMs’ capabilities in non-reasoning tasks where predefined evaluation datasets don’t exist. Our framework combines novel approaches in data generation, prompt-tuning, and LLM-based evaluation to demonstrate the potential of task-specific finetuning. This work provides practitioners with tools to effectively benchmark both SLMs and LLMs for practical applications, particularly in edge and private computing scenarios. Our implementation is available at: https://github.com/amazon-science/wraval . 1 Introduction Writing assistance tools powered by Language Models (LMs) are becoming increasingly prevalent on personal devices 32 ; 15 ; 23 . These applications typically use Small Language Models (SLMs) with fewer than 8B parameters due to memory and latency constraints of mobile devices. While SLMs may not match Large Language Models (LLMs) in complex reasoning tasks 7 ; 30 ; 37 , they are finding widespread adoption in focused applications like text rewriting. Despite this growing deployment of SLMs, existing benchmarks predominantly focus on evaluating LMs’ reasoning capabilities (e.g., MMLU 17 , SuperGLUE 33 ) 1 1 1 In addition to Math and STEM tasks (e.g. GPQA 28 ), and coding tasks (e.g. evalPlus 20 ). . This creates a significant gap: we lack standardized ways to evaluate SLMs on many of their typical use cases - including writing assistance tasks. We propose that writing assistance tasks represent a distinct category of language tasks where parameter efficiency can be achieved more readily than in reasoning tasks, as they primarily involve pattern recognition and stylistic transformation rather than complex logical operations across multiple inference steps. This efficiency difference may explain why SLMs can perform competitively on these tasks despite their parameter constraints. Figure 1: WRAVAL – Writing Assistant eVALuation pipeline. Our proposed framework (1) generates data with custom prompts at scale, (2) performs inference with models hosted on Bedrock / Ollama / Sagemaker Endpoints, (3) runs A bigger LLM-as-a-judge on all inputs-output pairs, (4) transforms the data into a format ready for human judges. A symbol hovers over each step of the pipeline to indicate that it is supported by either our own llm router (bottom left), or Sagemaker Groundtruth (bottom right). In this paper, we address this gap by introducing a novel evaluation framework for Writing Assistance (WA) tasks, which we define as single-turn text transformations guided by specific instructions. Our framework focuses on nine common rewrite instructions: casual, elaborate, emojify, improve, keypoints, professional, proofread, shorten and witty (see https://github.com/amazon-science/wraval for tone definitions). Below is an example professionalized message, and a verbatim output of the command wraval show-examples : ⬇ ========================================================================= Tone : professional | Model : qwen -3-4 B ========================================================================= Example 7545: Original : I was feelin ’ myself in that outfit , bruh , no lie . Rewrite : I felt confident in that outfit - no doubt about it . Score : 3.00 ---------------------------------------- Our framework departs from traditional static benchmarks by dynamically generating both evaluation data and assessments, enabling task-specific personalization. Using this approach, we demonstrate that SLMs are narrowing the performance gap with LLMs on writing assistance tasks. Additionally, our findings reveal unexpected limitations of LLMs in certain non-reasoning scenarios, challenging conventional assumptions about their superiority across all tasks (see Section 3.5 ). 2 Related Work SLMs on non-reasoning tasks Small Language Models (SLMs) have demonstrated increasing performance on non-reasoning tasks, maintaining coherence in English text generation and handling simpler tasks such as extraction and summarization effectively. For instance, models like Phi-1 have shown promising results in these capacities without extensive reasoning capabilities 11 ; 14 . This suggests a theoretical distinction between reasoning and non-reasoning language tasks, where the latter may require less computational complexity and can be effectively performed with fewer parameters. The parameter efficiency of non-reasoning tasks may be explained by their reliance on pattern recognition and application of learned linguistic conventions rather than multi-step logical inference. Evaluation Frameworks Robust evaluation of text improvement has been advanced by benchmarks such as EDITEVAL, which focuses on instruction-based assessments of modular text editing skills, including cohesion improvements and paraphrasing 10 . Additionally, Apple has developed a rigorous evaluation process using highly trained human graders with writing and comprehension expertise to assess summarization and naturalness of generated text across locales and tasks 15 . Evaluation Datasets Publicly available benchmark datasets play a pivotal role in evaluating language model capabilities across various tasks, including translation, summarization, and text improvements. For translation, the annually released Conference on Machine Translation (WMT) dataset provides multiple language pairs; our work adopts the WMT16 dataset specifically for German-to-English translation as an initial benchmark 8 . In summarization, widely used datasets such as Gigaword and SAMSum enable assessment of models’ ability to generate concise and coherent summaries 13 . For text improvements, datasets like EDITEVAL offer instruction-based evaluations, while MESSAGEREWRITEEVAL builds upon EDITEVAL by fine-tuning models on five distinct rewriting tasks—formalization, shortening, elaboration, paraphrasing, and proofreading—using task-specific instruction prompts and human judgment guidelines 10 ; 42 . Finetuning Parameter-efficient fine-tuning techniques such as LoRa have been successfully applied for multilingual summarization tasks, enabling effective model adaptation with fewer trainable parameters 36 . Reinforcement learning (RL) methods have been utilized for various text rewriting tasks including formalization, shortening, elaboration, paraphrasing, and proofreading. Such RL-driven finetuning has shown improvements in task-specific performance, with dedicated studies further focusing on optimizing proofreading via RL 42 ; 21 . Despite these resources, the limited scope of existing frameworks and datasets for non-reasoning tasks motivate the use of generated data and our own framework to further advance model training and evaluation in this domain. 3 Design This section is dedicated to the framework, and its associated experimental design. 3.1 Framework – WRAVAL: Writing Assistant eVALuation WRAVAL is an open-source library designed to evaluate Language Models (LMs) on Writing Assistance (WA) tasks, such as text professionalization and summarization. The framework implements a sequential workflow managed through a centralized table structure, where rows represent new data and columns represent workflow steps. Version control is maintained by time-stamping each table iteration, with flexible storage options for both local and cloud environments. The workflow comprises four primary steps illustrated in Figure 1 : data generation ( generate ), inference processing ( inference ), LLM-based evaluation ( llm judge ), and human judgment ( human judge ). The lm router and prompt directory components complement these core functions. Below, we detail each component before presenting recommended implementation guidelines. generate The generation step is responsible for generating a synthetic dataset with an LM. A single execution of this step can generate data either for a single tone or all possible tones. The prompt is of the form "generate data that could be turned more professional". We provide code to generate around a 100 (default value) examples with one prompt and

Read the complete text