What properties of reasoning supervision are associated with improved downstream model quality?

Mikołaj Langner, Dzmitry Pihulski, Jan Eliasz, Michał Rajkowski, Przemysław Kazienko, Maciej Piasecki, Jan Kocoń, Teddy Ferdinan · 2026

Read What properties of reasoning supervision are associated with improved downstream model quality? by Mikołaj Langner, Dzmitry Pihulski, Jan Eliasz, Michał Rajkowski, Przemysław Kazienko, Maciej Piasecki, Jan Kocoń, Teddy Ferdinan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13290.txt.

Introduction

Langner and colleagues ask whether a reasoning dataset can be judged before costly training runs. They propose intrinsic data measures and test them by tuning 8B and 11B models on variants of a Polish reasoning set. The measures track later model quality strongly, but the best predictors shift with scale. Small models need alignment measures for precision, while larger models gain from verbose traces and redundancy on hard tasks.

Catalog record

CREATORMikołaj Langner, Dzmitry Pihulski, Jan Eliasz, Michał Rajkowski, Przemysław Kazienko, Maciej Piasecki, Jan Kocoń, Teddy Ferdinan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36431
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.13290

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

What properties of reasoning supervision are associated with improved downstream model quality? Mikołaj Langner, Dzmitry Pihulski, Jan Eliasz, Michał Rajkowski, Przemysław Kazienko, Maciej Piasecki, Jan Kocoń, Teddy Ferdinan Abstract Validating training data for reasoning models typically requires expensive trial-and-error fine-tuning cycles. In this work, we investigate whether the utility of a reasoning dataset can be reliably predicted prior to training using intrinsic data metrics. We propose a suite of quantitative measures and evaluate their predictive power by fine-tuning 8B and 11B models on semantically distinct variants of a Polish reasoning dataset. Our analysis reveals that these intrinsic metrics demonstrate strong and significant correlations with downstream model performance. Crucially, we find that the predictors of utility are scale-dependent: smaller models rely on alignment-focused metrics to ensure precision, whereas larger models benefit from high redundancy, utilizing verbose traces to solve complex tasks. These findings establish a scale-aware framework for validating reasoning data, enabling practitioners to select effective training sets without the need for exhaustive empirical testing. 1 Introduction Explicit reasoning strategies [ 36 ] and specialized models [ 26 , 12 ] have transformed the capabilities of Large Language Models (LLMs). Consequently, fine-tuning on datasets enriched with reasoning traces has become the standard paradigm for imbuing these models with such skills. However, while the importance of high-quality data is universally acknowledged, the definition of quality for reasoning traces remains ambiguous. Currently, validating a reasoning dataset is an inefficient process that relies on post-hoc evaluation: researchers must fine-tune a model to discover if their data is effective. This training-as-validation approach is computationally prohibitive and unscalable. To democratize the development of robust reasoning models, the community requires objective and computable metrics that can validate the utility of training data before the expensive fine-tuning process begins. In this paper, we address this gap by establishing a link between intrinsic data characteristics and downstream model performance. We leverage a controlled set of Polish reasoning variants from our previous work [ 29 ] and the corresponding fine-tuned 8B and 11B models. By subjecting these known reasoning variants to a rigorous set of quantitative measurements, from linguistic complexity to semantic alignment, we determine which metrics serve as reliable predictors of a model’s final reasoning ability. Our analysis is guided by the following research questions: RQ1 : Is it feasible to validate the utility prior to fine-tuning? RQ2 : Which specific quantitative measures provide the most meaningful signal for validating training data quality? The contributions of this work are as follows: (1) a systematic evaluation of validation metrics for reasoning datasets, distinguishing between superficial statistics and deep semantic indicators; (2) a correlation analysis linking pre-training data scores with downstream performance; and, (3) a scale-aware framework for selecting reasoning data, allowing researchers to estimate model performance without incurring the full cost of training. 1.1 Related Work The precise utility of generated reasoning traces remains a subject of active debate. Shojaee et al. [ 32 ] argue that reasoning-augmented models often exhibit illusionary improvements, failing catastrophically on complex tasks while overthinking simple ones. Although Lawsen et al. [ 20 ] challenged these findings based on methodological discrepancies, the consensus remains that reasoning traces are not a guaranteed panacea. Furthermore, studies indicate that LLM reasoning often diverges from genuine logical inference [ 7 , 5 , 17 , 38 , 3 , 10 , 6 , 29 ] , with models frequently omitting premises or generating hallucinated reasoning steps that do not correlate with the accuracy of the final answer. Recent work has attempted to isolate specific attributes of reasoning data that drive performance, particularly sequence length. Jin et al. [ 16 ] posit that extending the length of reasoning, regardless of quality, can boost performance. In contrast, Wu et al. [ 39 ] demonstrate an inverted U-shaped relationship, suggesting that excessive length introduces error accumulation. Collectively, these conflicting findings suggest that neither length nor presence of reasoning alone are sufficient proxies for the utility of training data. Although prior work largely evaluates reasoning quality by analyzing model outputs, there is insufficient research on validation methods that evaluate reasoning data before committing computational resources to fine-tuning. Our work addresses this gap by correlating intrinsic data metrics with downstream performance established in our prior experiments. Figure 1 : We translated a subset of Mixture-of-Thoughts [ 13 , 23 , 27 , 2 ] into Polish, and split it into a training (MoT-PL) and evaluation set (MoT-PL-eval). Three additional variants of MoT-PL were created by paraphrasing only the reasoning part of each example: the Summarized style made the reasoning much more concise, the BabyThink style greatly simplified the reasoning, and the Lengthy style prolonged the reasoning. Afterwards, we fine-tuned PLLuM-8B-instruct and Bielik-11B-v2.6-Instruct on these datasets separately and evaluated them. 2 Experimental Setup 2.1 Datasets To rigorously evaluate the efficacy of pre-training validation metrics, four distinct reasoning datasets derived from the Polish Mixture-of-Thoughts (MoT-PL) were used. The original MoT-PL dataset was created by sampling approximately 32,000 examples from the English Mixture-of-Thoughts collection [ 13 ] and translating them into Polish using DeepSeek-V3 [ 9 ] . After filtering for errors and context length, the final dataset contained 22,571 examples spanning three domains: Mathematics (28%), Programming (17%), and Science (55%). To ensure the generated traces exhibited natural, human-like fluency rather than rigid machine translation artifacts, a randomly sampled subset of the DeepSeek-V3 outputs was manually verified by native Polish speakers. From this foundational dataset, we generated four semantically distinct variants to serve as our controlled variables (see Figure 1 ). These datasets, Detailed , Summarized , BabyThink , and Lengthy , share identical user prompts and final answers but differ significantly in the style, length, and semantic density of their reasoning traces. The general statistics of the dataset variants are shown in Table 1 . The variants were generated by automatic paraphrasing using DeepSeek-V3, resulting in the following profiles: • Detailed: The unmodified MoT-PL dataset, representing high-quality standard reasoning. The traces mimic the depth of the original English Mixture-of-Thoughts, serving as our control for standard reasoning density. • Summarized: A concise variant in which reasoning traces were compressed to retain essential logic while stripping stylistic fluff. This dataset tests the hypothesis that a higher information density correlates with efficiency. • BabyThink: A variant paraphrased into "childlike" language. Rather than merely reducing statistical readability, the prompt intentionally obfuscates specific details and calculations with vague filler. The original train of thought and structure are strictly preserved to avoid injecting artificial hallucinations or new reasoning fallacies. • Lengthy: An artificially prolonged variant designed to be approximately twice as long as the Detailed version. It preserves the original logic but introduces verbosity, allowing us to test if metrics favoring longer chains are misleading. Table 1 : Statistical profile of the dataset variants used for metric validation. All variants share identical question/answer pairs; variations occur strictly within the reasoning trace. The first value of a token count comes from using the PLLuM-8B-instruct tokenizer, while the second value comes from using the Bielik-11B-v2.6-Instruct tokenizer. Metric (Avg.) Detailed Summarized BabyThink Lengthy Reasoning Tokens 2,613~2,918 399~452 3,729~4,000 9,930~10,902 Reasoning Chars 6,334 965 8,991 17,369 Total Seq. Tokens 3,288~3,685 1,073~1,219 4,404~4,767 10,604~11,669 Total Seq. Chars 8,090 2,721 10,747 25,938 All examples exceeding the context window

Read the complete text