Reward Modeling for Scientific Writing Evaluation
Furkan Şahinuç, Subhabrata Dutta, Iryna Gurevych · 2026
Read Reward Modeling for Scientific Writing Evaluation by Furkan Şahinuç, Subhabrata Dutta, Iryna Gurevych (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-11374.txt.
Introduction
Furkan Sahinuc, Subhabrata Dutta, and Iryna Gurevych grade scientific writing by machine. Their 2026 preprint builds reward models for that task. Scientific writing is expert work with strict demands. It needs deep domain knowledge and sharp reasoning at once.
Making scientific text by machine has been widely studied. Grading such text is still an open, hard problem. A reward model scores drafts so systems can learn from scores. Scores must track task specs, not just fluent style. The team tackles what a good reward for science prose is. Sound grading could lift AI help for authors and reviewers. It could also flag weak claims dressed in fine words.
Catalog record
| CREATOR | Furkan Şahinuç, Subhabrata Dutta, Iryna Gurevych |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 106 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-11374
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reward Modeling for Scientific Writing Evaluation Furkan Şahinuç, Subhabrata Dutta, Iryna Gurevych Abstract Scientific writing is an expert-domain task that demands deep domain knowledge, task-specific requirements and reasoning capabilities that leverage the domain knowledge to satisfy the task specifications. While scientific text generation has been widely studied, its evaluation remains a challenging and open problem. It is critical to develop models that can be reliably deployed for evaluating diverse open-ended scientific writing tasks while adhering to their distinct requirements. However, existing LLM-based judges and reward models are primarily optimized for general-purpose benchmarks with fixed scoring rubrics and evaluation criteria. Consequently, they often fail to reason over sparse knowledge of scientific domains when interpreting task-dependent and multi-faceted criteria. Moreover, fine-tuning for each individual task is costly and impractical for low-resource settings. To bridge these gaps, we propose cost-efficient, open-source reward models tailored for scientific writing evaluation. We introduce a two-stage training framework that initially optimizes scientific evaluation preferences and then refines reasoning capabilities. Our multi-aspect evaluation design and joint training across diverse tasks enable fine-grained assessment and robustness to dynamic criteria and scoring rubrics. Experimental analysis shows that our training regime strongly improves LLM-based scientific writing evaluation. Our models generalize effectively across tasks and to previously unseen scientific writing evaluation settings, allowing a single trained evaluator to be reused without task-specific retraining. We make our code 1 1 1 GitHub: UKPLab/arxiv2026-expert-eval-rm and data 2 2 2 Data: TUdatalib publicly available. Figure 1: Example demonstration of how we formalize scientific writing evaluation and the outputs of the review utility evaluation task. Vanilla LLM-based judges fail to properly reason over the task-specific evaluation criteria and provided examples. Contradictory statements are highlighted in different colors. In contrast, our SciRM model successfully incorporates the given criteria and examples into its reasoning process and correctly evaluates the scientific artifact. 1 Introduction Figure 2: Overview of SciRM and SciRM-Ref training and testing pipeline. Diverse scientific artifacts are used to construct training data with multiple evaluation aspects and scoring rubrics (see Section 3.1 for details). Models are trained via GRPO in two stages to optimize task specifications and reasoning capabilities, and are evaluated on both seen and unseen scientific writing evaluation tasks. Due to the strong text generation capabilities of large language models (LLMs), their application to scientific text generation, such as related work generation, review generation, and revising papers, has gained increasing attention recently 23 ; 25 ; 1 . However, without proper evaluation, it is difficult to assess the accuracy and reliability of the generated texts. Therefore, the limitations in evaluation can block the entire development pipeline. Since scientific writing tasks have diverse and task-specific requirements, developing appropriate evaluation frameworks is a challenging problem. Training evaluators for each individual task is costly and, in some cases, infeasible due to limited data availability. LLM-as-a-judge approaches 28 ; 47 are the most widely adopted evaluation paradigms in scientific writing tasks. However, they often fail to reason over the given domain knowledge and task-specific preferences (see Figure 1 ). This motivates the need for mechanisms that allow models to reason over and remain grounded in explicit evaluation guidelines (aka, constitution ) at inference time. Inference-time adaptability is a major challenge for existing approaches like Constitutional AI 4 , which internalize fixed constitution during training and therefore cannot be readily applied to a diverse set of evaluation tasks. This rigidity is problematic while evaluating scientific text generations, as evaluation guidelines can be different, even contradictory, across different aspects, tasks, or domains. To improve the reasoning capabilities of LLM-based judges, training reward reasoning models has recently gained popularity 3 ; 14 ; 6 ; 7 ; 41 . However, existing reward models are primarily optimized for improving performance on community-standard benchmarks, such as mathematical reasoning, instruction-following, and human-preference modeling for coding, helpfulness, and safety 22 ; 31 ; 11 . Therefore, they fall short of capturing the nuanced requirements of scientific writing evaluation. In addition, the vast majority of reward models encode task preferences in a pairwise manner, which prevents independent assessment of text quality based on explicit task-specific criteria. Another drawback of such models is that they are optimized for fixed scoring rubrics and criteria. Scientific tasks have unique characteristics; each requires domain-specific expertise and evaluation dynamics that differ from those of open-ended creative writing tasks 5 . Even for the same scientific artifact, it is possible to evaluate from multiple aspects requiring different criteria and rubrics. However, current reward models experience performance degradation when they are applied to other tasks with evaluation rubrics that differ from those seen during training 46 . On the other hand, 35 offers insights on multiple-aspect evaluation for expert-domain tasks. However, their approach is limited to a single task, and their most accurate pipeline relies on proprietary LLMs, which restricts scalability and hinders generalization across diverse scientific writing tasks. In this work, we adapt reward model training strategies to enhance LLMs’ capabilities regarding what to evaluate and how to evaluate scientific writing tasks. Concretely, we design reward models that are conditioned on an explicit evaluation constitution—a structured description of criteria and label space—present during both training and inference. Furthermore, we introduce a two-stage optimization process in which models not only learn to follow the constitution but also reflexively reinterpret it to correct and stabilize their own reasoning. This process involves a joint optimization of in-context preference-following and reasoning abilities, which 21 identifies as a missing piece in modern LMs trained using RL. Contributions and Findings: We introduce cost-efficient reward models, SciRM and SciRM-Ref , specifically designed for scientific writing evaluation (C1) . We employ two-stage reinforcement learning to optimize the models for (1) scientific writing evaluation preferences and (2) reasoning abilities to better comprehend the given evaluation criteria, enabling models to explicitly reason over and faithfully adhere to dynamically specified evaluation rules. (C2) . Rather than producing a single aggregated score, our models evaluate scientific artifacts across multiple aspects. This approach enhances both reliability and interpretability of our evaluation (C3) . We curate and process datasets from diverse sources and jointly train our models across multiple tasks to (1) improve robustness against varying scoring rubrics and (2) enhance our models’ generalization capabilities (C4) . We illustrate the overview of our pipeline in Figure 2 . We finally test our models on four different scientific writing tasks: related work sections, paper reviews, novelty summary assessments, and paper revisions based on instructions, each with distinct evaluation aspects and scoring rubrics. Experimental results show that our two-stage training scheme substantially boosts the LLMs’ scientific writing evaluation performance (F1) . In particular, our second training stage leads to improvements in tasks requiring strong reasoning capabilities (F2) . Furthermore, our models outperform baseline models on tasks that are not included in the training, indicating our model’s strong generalization and scaling capabilities (F3) . 2 Related Work 2.1 Scientific Writing Evaluation Utilizing LLMs as direct evaluators is one of the most intuitive approaches for scientific writing evaluation due to their flexibility in being prompted with various task settings without additional training overhead 28 ; 47 . However, prior work has shown that vanilla LLM-as-a-judge setups are prone to systematic biases and failures in domain-grounded reasoning 24 ; 39 ; 12