Value-Based Pre-Training with Downstream Feedback
Shuqi Ke, Giulia Fanti · 2026
Read Value-Based Pre-Training with Downstream Feedback by Shuqi Ke, Giulia Fanti (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22108.txt.
Introduction
Shuqi Ke and Giulia Fanti ask whether a little verified feedback can steer a very expensive process. Foundation models are pretrained with fixed proxy goals, such as predicting the next token. Those proxies can waste compute on skills that downstream users never need.
Their 2026 preprint introduces value-based pretraining with downstream feedback. A small amount of verified information about the goal guides where the pretraining effort goes, measured by value rather than by the proxy alone. If steering works, the huge cost of pretraining could be aimed better. It could target capabilities people actually want. It would not be spread evenly over everything a model might ever be asked to do.
Catalog record
| CREATOR | Shuqi Ke, Giulia Fanti |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-22108
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Value-Based Pre-Training with Downstream Feedback Shuqi Ke, Giulia Fanti Abstract Can a small amount of verified goal information steer the expensive self-supervised pretraining of foundation models? Standard pretraining optimizes a fixed proxy objective (e.g., next-token prediction), which can misallocate compute away from downstream capabilities of interest. We introduce V-Pretraining: a value-based, modality-agnostic method for controlled continued pretraining in which a lightweight task designer reshapes the pretraining task to maximize the value of each gradient step. For example, consider self-supervised learning (SSL) with sample augmentation. The V-Pretraining task designer selects pretraining tasks (e.g., augmentations) for which the pretraining loss gradient is aligned with a gradient computed over a downstream task (e.g., image segmentation). This helps steer pretraining towards relevant downstream capabilities. Notably, the pretrained model is never updated on downstream task labels; they are used only to shape the pretraining task. Under matched learner update budgets, V-Pretraining of 0.5B–7B language models improves reasoning (GSM8K test Pass@1) by up to 18% relative over standard next-token prediction using only 12 12 % of GSM8K training examples as feedback. In vision SSL, we improve the state-of-the-art results on ADE20K by up to 1.07 mIoU and reduce NYUv2 RMSE while improving ImageNet linear accuracy, and we provide pilot evidence of improved token efficiency in continued pretraining. 1 Introduction Figure 1: Value-Based Pretraining with Downstream Feedback. Today, the learner \theta trains on unlabeled data using a proxy objective L_{\mathrm{pre}} , for a frozen pretraining task. In V-Pretraining, a small task designer \phi is trained on a small feedback set of verifiable downstream tasks with predefined value functions, but never updates the learner on downstream labels. \phi thus reshapes the pretraining target (or views) so that the induced SSL update aligns with downstream improvement, calculated via the value function. Relative to current pretraining methods, V-Pretraining adds the components in the left blue box. The era of blind scaling that improves models primarily by scaling proxy-objective pretraining is showing signs of diminishing returns ( 30 ; 26 ; 22 ) . Yet foundation models are still trained in a remarkably undirected way: we minimize a static self-supervised proxy loss on massive, weakly curated data, and hope the capabilities we care about (reasoning, dense perception, tool use, world modeling) emerge as a byproduct. In language, the proxy is next-token prediction ( 7 ; 35 ; 54 ) ; in vision, it is self-supervised reconstruction or representation learning under augmentations ( 11 ; 21 ; 2 ; 46 ) . While this recipe scales, it functions as an open-loop system and learns from a “static world”: the optimization trajectory is fixed at the start, ignoring whether intermediate steps actually align with complex human goals. This open-loop nature can lead to sample inefficiency in pretraining. Unlike humans, who utilize closed-loop feedback to rapidly correct errors and master tasks, models blindly consume trillions of tokens without corrective guidance. Current pipelines inject feedback mostly after pretraining via supervised fine-tuning or preference optimization ( 12 ; 36 ; 40 ) . These stages are effective, but they arrive late. By the time downstream feedback is applied, the representation has already been shaped by millions of proxy-gradient steps that were agnostic to the target behavior. To break the ceiling of blind scaling, we ask: can we introduce scalable supervision into pretraining, turning an open-loop process into a controlled trajectory toward what we actually want? We introduce V-Pretraining: V alue-based Pre-Training with downstream feedback , a framework for controlled pretraining . Standard pretraining fixes the unlabeled stream and a proxy task construction (e.g., one-hot next-token targets in language, or a fixed augmentation pipeline in vision) and optimizes the resulting proxy loss. We keep the unlabeled stream and learner training budget fixed, but add a lightweight task designer trained on a small labeled verification (value) set for the capability of interest (e.g., GSM8K for reasoning, ADE20K/NYUv2 for dense vision). Crucially, the verification set is used only as an evaluator: the learner is never updated on verification labels. Instead, the task designer reshapes the pretraining target (the supervision signal inside predictive learning) so that the learner’s next unlabeled update is predicted to be more valuable for the target capability. In language, the designer replaces one-hot next-token labels with adaptive soft targets supported on the learner’s top- K candidates. In vision SSL, it replaces a fixed augmentation pipeline with instance-wise learned views optimized for transfer, especially dense prediction. Directly optimizing the task designer for downstream performance is computationally prohibitive: it is a bilevel problem that would require differentiating through long pretraining trajectories ( 31 ; 18 ) . As a result, prior efforts to design task-aware SSL methods were largely tailored to specific domains and tasks, which allowed them to avoid this computational bottleneck ( 57 ; 51 ; 45 ) . A key insight of our work is showing how to efficiently generalize task-aware pretraining (including SSL) to different tasks and modalities by defining the value of a pretraining step via an influence-style first-order estimate: the alignment between proxy and downstream gradients ( 27 ; 37 ) . This yields differentiable meta-updates for the task designer while leaving the learner’s pretraining loop essentially unchanged. Because V-Pretraining intervenes only through target/view construction, it can be layered on top of diverse pretraining objectives (e.g., next-token prediction, masked modeling, and joint-embedding SSL) without changing the learner architecture or optimizer. This makes V-Pretraining largely orthogonal to advances in scaling, data mixture/curriculum design, and post-training alignment, and in principle combinable with them. Across language and vision, value-based pretraining turns small verified feedback into measurable gains in the expensive unlabeled phase. In language, continued pretraining of Qwen1.5 models on a math corpus improves GSM8K Pass@1 by 2-14% across 0.5B/4B/7B using only 12\% GSM8K training examples for feedback and without updating the learner on GSM8K labels . In vision, dense feedback improves segmentation and depth while maintaining or improving ImageNet linear accuracy. Further, these improvements do not come at the expense of generalization to other tasks. Contributions. We make four contributions. (1) V-Pretraining: a novel framework for directed pretraining with downstream feedback: we present a principled formulation of controlled pretraining as goal-directed target or view design, separating a large learner trained only on unlabeled data from a lightweight controller trained on a small labeled verification set. (2) A scalable learning rule for task design: we introduce an influence-style first-order value objective based on proxy–downstream gradient alignment that avoids differentiating through long pretraining trajectories. (3) Efficient instantiations across modalities: we instantiate the framework for natural language (adaptive top- K soft targets) and vision (instance-wise learned views) without changing the learner’s underlying pretraining loop. (4) Compute-matched evidence and diagnostics: we empirically show that under matched learner update budgets, the V-Pretraining framework increases downstream value per pretraining step for two modalities (vision and language) in various settings. We support our claims with extensive ablations (random feedback, smoothing, self-distillation) and controllability diagnostics (token-efficiency pilot; Pareto tradeoffs in vision). Together, these results suggest that a small amount of indirect downstream feedback can act as a scalable form of weak-to-strong supervision during pretraining ( 8 ) , improving target capabilities under fixed compute budgets, without harming model generalization. 2 Preliminaries and Related Work 2.1 Pretraining as Predictive Learning We unify modern