Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capability

Qihan Ren, Peng Wang, Ruikun Cai, Shuai Shao, Dadi Guo, Yuejin Xie, Yafu Li, Quanshi Zhang, Xia Hu, Jing Shao, Dongrui Liu · 2026

Read Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capability by Qihan Ren, Peng Wang, Ruikun Cai, Shuai Shao, Dadi Guo, Yuejin Xie, Yafu Li, Quanshi Zhang, Xia Hu, Jing Shao, Dongrui Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-06628.txt.

Introduction

Ren and ten colleagues revisit the claim that supervised fine-tuning only memorizes while reinforcement learning generalizes. For reasoning fine-tuning with long chain-of-thought supervision, cross-domain transfer proves conditional, shaped jointly by training dynamics, data, and base-model strength. Short training can mislead, since performance first dips then recovers, while verified long traces help and stronger models pick up transferable habits like backtracking. The catch is asymmetry: reasoning improves while safety slips, so the question becomes under what conditions and at what cost transfer happens.

Catalog record

CREATORQihan Ren, Peng Wang, Ruikun Cai, Shuai Shao, Dadi Guo, Yuejin Xie, Yafu Li, Quanshi Zhang, Xia Hu, Jing Shao, Dongrui Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE118925
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.06628

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capability Qihan Ren, Peng Wang, Ruikun Cai, Shuai Shao, Dadi Guo, Yuejin Xie, Yafu Li, Quanshi Zhang, Xia Hu, Jing Shao, Dongrui Liu Abstract A prevailing narrative in LLM post-training holds that supervised fine-tuning (SFT) memorizes while reinforcement learning (RL) generalizes. We revisit this claim for reasoning SFT with long chain-of-thought (CoT) supervision and find that cross-domain generalization is not absent but conditional , jointly shaped by optimization dynamics, training data, and base-model capability. Some reported failures are under-optimization artifacts: cross-domain performance first degrades before recovering and improving with extended training (a dip-and-recovery pattern), so short-training checkpoints can underestimate generalization. Data quality and structure both matter: low-quality solutions broadly hurt generalization, while verified long-CoT traces yield consistent cross-domain gains. Model capability is essential: stronger models internalize transferable procedural patterns (e.g., backtracking) even from a toy arithmetic game, while weaker ones imitate surface verbosity. This generalization is asymmetric, however: reasoning improves while safety degrades, reframing the question from whether reasoning SFT generalizes to under what conditions and at what cost. † † footnotetext: ∗ Equal contribution. Work done during an internship at Shanghai Artificial Intelligence Laboratory, supervised by Dongrui Liu † Corresponding author 1 Introduction A prevailing narrative in Large Language Model (LLM) post-training is that supervised fine-tuning (SFT) improves in-domain performance but is prone to memorization, whereas reinforcement learning (RL) generalizes better. 4 established the influential “SFT memorizes, RL generalizes” framing on synthetic tasks, and subsequent work reported similar patterns in more realistic settings such as math reasoning ( 9 ) . This narrative has motivated considerable efforts in understanding RL’s advantage over SFT ( 3 ; 33 ; 14 ; 45 ) and in algorithmic modifications to the SFT objective itself ( 36 ; 46 ) . However, the experimental conditions under which this narrative was established vary considerably. For instance, 4 trained without long chain-of-thought (CoT) supervision; 9 trained for relatively short epochs; 36 used data with uneven response quality; and many studies, constrained by compute or time, used small or early-generation base models without explicitly considering model capability as a factor. Separately, many SFT-vs-RL comparisons focus on retention (whether fine-tuning degrades existing abilities) rather than on the acquisition of new generalization, and often start from instruction-tuned models where confounds from alignment are hard to disentangle ( 3 ; 14 ) . These entangled differences in optimization, data, model capability, and starting checkpoint make it unclear whether reported generalization failures are intrinsic to SFT or merely artifacts of particular experimental conditions. This question is particularly pressing for reasoning SFT , where models are often trained on long CoT traces. Long-CoT traces are different in data structure, harder to fit than short-form targets, and more demanding of model capability. This makes reasoning SFT a setting where optimization, data, and model factors are critical and can be more clearly studied. In this work, we show that generalization in reasoning SFT is not an inherent property of the training objective itself, but a conditional phenomenon jointly shaped by optimization dynamics , training data , and model capability . We systematically vary each factor to identify when and how reasoning SFT generalizes across domains, and what trade-offs arise when it does. We adopt math-only reasoning SFT on pretrained base models as a testbed, a setting that provides easily verifiable responses and has direct comparison with prior studies ( 9 ; 36 ) . We then evaluate generalization across in-domain math reasoning, OOD reasoning (e.g., code, science), general capabilities (e.g., instruction following), and safety. Our findings are as follows: Figure 1: Conceptual framework: generalization in reasoning SFT is a conditional property jointly shaped by optimization dynamics, training data, and model capability. • Apparent non-generalization may be an under-optimization artifact. Generalization performance exhibits a dip-and-recovery pattern during long-CoT SFT: it first degrades before recovering and eventually surpassing the base model, suggesting that short-training checkpoints can systematically underestimate SFT’s generalization potential. Further analysis shows that long-CoT data benefits more from repeated exposure than from one-pass coverage under matched compute, and that pronounced overfitting symptoms appear only under aggressive training schedules. This indicates that under-optimization may be a more prevalent risk than over-optimization in this regime. • Training data matters for generalization. Data quality matters: SFT on low-quality solutions can broadly hurt performance, whereas verified long-CoT traces yield consistent cross-domain gains. Data structure also matters: on strong base models, long-CoT traces from a toy arithmetic game (Countdown) can improve performance on several reasoning benchmarks (e.g., math, code, science), and can even outperform a no-CoT dataset with diverse math problems. This suggests that procedural patterns in long CoT traces, such as backtracking and verification, may be what generalizes beyond math content itself. • Generalization requires sufficient model capability. Given identical long-CoT data, stronger models show broad generalization across domains, while weaker models show marginal or negative gains (even on in-domain math) and tend to produce prolonged responses. This suggests weaker models may imitate the surface form of reasoning (e.g., verbosity) without internalizing the patterns that drive cross-domain generalization. • Generalization is asymmetric. Despite broad gains, long-CoT SFT weakens safety, consistent with findings on self-jailbreaking in reasoning models ( 41 ; 25 ) . After training, models self-rationalize during thinking (e.g., “for educational purposes”) and eventually generate harmful content. In our controlled comparison, safety drop is much larger with CoT than without CoT, suggesting that this degradation is driven by the procedural patterns rather than domain content. These results suggest that the question “ does SFT generalize?” is under-specified. Generalization is not a fixed property of the SFT objective but varies substantially with optimization sufficiency, data quality and structure, and base model capability. Conclusions drawn when any of these factors is lacking (e.g., evaluating early checkpoints, training on low-quality data, or using weak base models) may mistake artifacts of the experimental setup for inherent limitations of SFT. A more productive framing is to ask under what conditions reasoning SFT generalizes, and our study is an early effort to plot this conditional landscape. 2 Experimental Setup This section describes the default experimental setup. Specific variations in training schedule, data, and base model are introduced in later sections where they become relevant. 2.1 Model, dataset, and training protocol Our main experiments use Qwen3-14B-Base and Qwen3-8B-Base ( 39 ) as base models, and include InternLM2.5-20B-Base ( 1 ) and Qwen2.5 base models ( 40 ) to assess whether the observed trends extend beyond one model family. All these models are pretrained checkpoints before instruction tuning, which minimizes confounds from alignment or preference optimization. Subsequent sections additionally examine smaller models (e.g., 4B and 1.7B) to study the role of model capability (Section 5 ). Our default training dataset, Math-CoT-20k , consists of 20,480 math reasoning examples with long CoT. The queries are sampled from the default subset of OpenR1-Math-220k ( 23 ) , and the responses are generated by Qwen3-32B with thinking enabled. Each response contains a thinking process followed by step-by-step final summary and answer. We generate multiple responses per query and use math-verify ( 10 ) to retain only those with correct answers. The maximum response length is set to 16,384 tokens. Please see App. B.1 for

Read the complete text