Transferability for General Reasoning: An Automated Curriculum for Multi-Domain RLVR
Yongjin Yang, Jiarui Liu, Yinghui He, Lechen Zhang, Bernhard Schölkopf, Zhijing Jin, Jinesis Lab, University of Toronto & Vector Institute, Carnegie Mellon University, Princeton University, University of Illinois Urbana-Champaign, ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, EuroSafeAI {yjyang,zjin}@cs.toronto.edu · 2026
Read Transferability for General Reasoning: An Automated Curriculum for Multi-Domain RLVR by Yongjin Yang, Jiarui Liu, Yinghui He, Lechen Zhang, Bernhard Schölkopf, Zhijing Jin, Jinesis Lab, University of Toronto & Vector Institute, Carnegie Mellon University, Princeton University, University of Illinois Urbana-Champaign, ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, EuroSafeAI {yjyang,zjin}@cs.toronto.edu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25178.txt.
Introduction
Training a reasoning model across maths, programming, and science at once raises a scheduling question: how often should each domain be sampled? Yongjin Yang and five colleagues note that skills transfer unevenly between domains, while existing curricula adapt only to where the model is currently improving.
Their Transfer Aware Curriculum is a bandit style scheduler that favours domains whose updates help the rest of the suite. It reuses signals the training run already produces. Per domain advantages capture local learnability. Projected gradients from the current step estimate cross domain transfer by gradient alignment. The extra cost stays under one percent. Across a six domain suite, the method posts the best average accuracy on both Qwen3-1.7B and Llama3.2-3B. It beats random sampling, a hand tuned schedule, and a learnability only bandit, the last by ten percent in relative terms.
Catalog record
| CREATOR | Yongjin Yang, Jiarui Liu, Yinghui He, Lechen Zhang, Bernhard Schölkopf, Zhijing Jin, Jinesis Lab, University of Toronto & Vector Institute, Carnegie Mellon University, Princeton University, University of Illinois Urbana-Champaign, ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, EuroSafeAI {yjyang,zjin}@cs.toronto.edu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 84 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.25178
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Transferability for General Reasoning: An Automated Curriculum for Multi-Domain RLVR Yongjin Yang, Jiarui Liu, Yinghui He, Lechen Zhang, Bernhard Schölkopf, Zhijing Jin, Jinesis Lab, University of Toronto & Vector Institute, Carnegie Mellon University, Princeton University, University of Illinois Urbana-Champaign, ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, EuroSafeAI {yjyang,zjin}@cs.toronto.edu Abstract Reinforcement learning with verifiable rewards (RLVR) has been extended from single-domain training to multi-domain reasoning suites spanning mathematics, programming, and science. However, the training curriculum (how often each domain is sampled) is typically fixed or hand-tuned, even though reasoning skills transfer unevenly across domains. Existing learnability-based curricula adapt to where the policy is currently improving, but are blind to whether a gradient step on the selected domain benefits the remaining domains. In this paper, we propose Transfer-Aware Curriculum ( TAC ), a bandit-style online curriculum that prioritizes domains whose updates broadly benefit the rest of the training suite. TAC repurposes signals already produced by RL training: per-domain advantages capture local learnability, and projected gradients, taken from the GRPO step being computed, estimate cross-domain transferability via gradient-geometry alignment, at negligible cost ( < < 1% wall-clock overhead). Across a six-domain reasoning suite, TAC achieves the best macro-averaged accuracy on both Qwen3-1.7B and Llama3.2-3B , outperforming proportional random sampling, a hand-designed schedule, and a learnability-only bandit, and improving over the last of these by up to 2.8 points (10% relative). Ablations show performance degrades sharply when the transferability term is removed, and TAC remains robust on imbalanced training mixtures where learnability-only curricula over-commit to dominant domains. Our findings establish cross-domain transferability as a key signal for curriculum design in multi-domain RLVR. 1 Introduction Reinforcement learning with verifiable rewards (RLVR) has become a central tool for improving the reasoning capabilities of large language models (LLMs), producing substantial gains on benchmarks where correctness can be checked automatically [ 17 , 50 ] . Motivated by these successes, recent work has extended RL training beyond single-domain setups toward broader multi-domain reasoning suites spanning domains such as mathematics, programming, and science [ 36 , 11 ] . The goal of this line of work is a single policy that reasons competently across heterogeneous tasks rather than one specialized in a narrow slice. Building a strong multi-domain reasoner, however, is not merely a matter of pooling data from every domain. Recent analyses show that reasoning training transfers unevenly across tasks, with effects depending strongly on the source domain [ 20 , 30 , 11 ] . We observe the same pattern in our own setting (Figure 1(a) ): RL-training on different single domains yields markedly different transfer profiles across the remaining domains. For example, given the same training budget, RL on table improves simulation accuracy by 14.6 percentage points, while RL on math improves it by only 5.0. Methodological responses have so far focused on optimization- and loss-level interventions: per-domain gradient alignment [ 31 ] or per-task loss reweighting for multi-task GRPO [ 45 ] . The orthogonal question of which domain to sample at each step , and how that choice should depend on cross-domain transferability, has received far less attention. Existing curricula for LLM reasoners [ 8 , 55 , 22 ] operate at the sampling level, but answer only half of this question. They prioritize domains whose on-policy advantages [ 8 , 55 ] or reward variances indicate active learning [ 22 ] , telling us which domain the policy can presently learn from , but not whether a gradient step on that domain also benefits the remaining training domains . A highly learnable domain may be narrowly scoped, so that gains on it fail to transfer; a less obviously learnable domain may nonetheless yield updates whose direction broadly improves the policy. A curriculum blind to cross-domain transfer can over-commit to locally rich but globally narrow domains, leaving a large share of the achievable cross-domain improvement on the table. (a) Cross-domain transfer matrix. (b) TAC vs. baseline and prior-work selection signals. Figure 1 : (a) Each cell shows the accuracy gain (pp) on a target domain after RL-training on a single source domain (1,000 queries, two epochs). Off-diagonal transfer varies sharply by source. The bottom rows show that TAC improves over Random across all six domains. (b) TAC combines a learnability signal with a gradient-cosine transferability signal that captures whether updates on a domain align with those of the other domains, mixed into per-arm scores via a single coefficient \beta . Contributions. In this paper, we propose Transfer-Aware Curriculum ( TAC ), a curriculum for multi-domain RL that introduces cross-domain transferability as a first-class signal for domain selection. While prior curricula prioritize domains the policy can presently learn from, TAC additionally asks whether a gradient step on a selected domain benefits the remaining training domains, and steers sampling accordingly. Figure 1(a) previews the result: uniform multi-domain sampling ( Random ) already outperforms any single-source curriculum on every target, and TAC widens this gap further across all six domains. The two signals come apart in practice: the most learnable domain is often not the most transferable . Notably, math , the domain RLVR leans on most, ranks among the least transferable in our suite, and TAC down-weights it accordingly. Our specific contributions are as follows: 1. Transfer-Aware Bandit Curriculum. We formulate multi-domain RL training as a multi-armed bandit over domains (§ 2.3 ), whose per-arm feedback combines an on-policy learnability term (§ 3.1 ) derived from GRPO advantages with a novel gradient-based transferability term, mixed via a single coefficient \beta (§ 3.3 ). The resulting method, TAC , jointly prioritizes domains that are currently learnable and broadly beneficial to the rest of the training set. 2. Self-Supervised Transferability Signal. We introduce a gradient-geometry estimator of cross-domain transferability (§ 3.2 ). At each training step, we maintain a per-domain exponential moving average of projected gradients; every K_{c} steps, the pairwise cosine similarities between these EMAs serve as the transferability signal fed back to the bandit. The signal is computed entirely from gradients already produced by RL training, requiring no held-out probes, extra rollouts, or oracle annotations, and adapts with the policy as it evolves. 3. Empirical Gains on Multi-Domain Reasoning. Across a six-domain reasoning suite spanning mathematics, programming, logic, simulation, tables, and stem, TAC consistently outperforms proportional sampling, a hand-designed math-to-others schedule, and a learnability-only bandit, improving macro-averaged accuracy across 14 evaluation benchmarks by 1.6–2.8 points (up to 10% relative) on Qwen3-1.7B and Llama3.2-3B at < 1% wall-clock overhead (§ 4 , Appendix C.2 ). Ablations isolate the transferability term, and TAC remains robust under data-budget skew where learnability-only curricula over-commit to dominant domains. 2 Preliminaries 2.1 Multi-Domain Reinforcement Learning for Reasoning Let \mathcal{D}=\{D_{1},\dots,D_{M}\} be a collection of reasoning domains, where each D_{m} consists of prompt–answer pairs (x,y^{*}) with verifiable targets. Given a query x , the policy \pi_{\theta} generates a response o\sim\pi_{\theta}(\cdot\mid x) , evaluated by a sparse correctness reward r(o,y^{*})=\mathbb{I}[\mathrm{Ans}(o)=y^{*}] . Multi-domain RL seeks a single policy maximizing expected reward under a sampling distribution \mu\in\Delta^{M-1} over domains: J(\theta;\,\mu)=\sum_{m=1}^{M}\mu_{m}\;\mathbb{E}_{(x,y^{*})\sim D_{m},\,o\sim\pi_{\theta}(\cdot\mid x)}\bigl[r(o,y^{*})\bigr]. (1) 2.2 Group Relative Policy Optimization (GRPO) We optimize \theta with Group Relative Policy Optimization (GRPO; [ 50 ] ). For a query x , GRPO draws K rollouts \{o^{(k)}\}_{k=1}^{K} from the old