VOSR: A Vision-Only Generative Model for Image Super-Resolution Rongyuan Wu, Lingchen Sun, Zhengqiang Zhang, Xiangtao Kong, Jixin Zhao, Shihao Wang, Lei Zhang, [0.5em] The Hong Kong Polytechnic University Abstract Most of the recent generative image super-resolution (SR) methods rely on adapting large text-to-image (T2I) diffusion models pretrained on web-scale text-image data. While effective, this paradigm starts from a generic T2I generator, despite that SR is fundamentally a low-resolution (LR) input-conditioned image restoration task. In this work, we investigate whether an SR model trained purely on visual data can rival T2I-based ones. To this end, we propose VOSR , a V ision- O nly generative framework for SR . We first extract semantically rich and spatially grounded features from the LR input using a pretrained vision encoder as visual semantic guidance. We then revisit classifier-free guidance for training generative models and show that the standard unconditional branch is ill-suited to restoration models trained from scratch. We therefore replace it with a restoration-oriented guidance strategy that preserves weak LR anchors. Built upon these designs, we first train a multi-step VOSR model from scratch and then distill it into a one-step model for efficient inference. VOSR requires less than one-tenth of the training cost of representative T2I-based SR methods, yet in both multi-step and one-step settings, it achieves competitive or even better perceptual quality and efficiency, while producing more faithful structures with fewer hallucinations on both synthetic and real-world benchmarks. Our results, for the first time, show that high-quality generative SR can be achieved without multimodal pretraining. The code and models can be found at https://github.com/cswry/VOSR . † † footnotetext: * Equal contribution. \dagger Corresponding author. This research is supported by the PolyU-OPPO Joint Innovative Research Center. Figure 1 : Comparison of VOSR with existing generative SR methods in terms of performance , efficiency , and training cost . Blue/orange colors denote T2I-based and vision-only methods, circles/triangles denote multi-step and one-step models. Performance is measured on RealSR 2 , and efficiency is measured at 512\times 512 resolution using official repositories. VOSR achieves competitive or better perceptual quality than many T2I-based SR methods in both multi-step and one-step settings, while clearly outperforming prior vision-only methods. Its multi-step variant is substantially more efficient than existing T2I-based methods, and its one-step variant remains comparable to recent one-step T2I systems. Measured by the total number of training pixels consumed, VOSR requires only about one-tenth of the training cost of representative T2I-based SR methods. For fairness, we count only the pretraining cost of the core diffusion modules; reused components such as the VAE and semantic encoders are excluded. 1 Introduction Image super-resolution (SR) is a fundamental problem in low-level vision, aiming to reconstruct a high-resolution (HR) image from its degraded low-resolution (LR) observation. Unlike the free-form image synthesis problem, SR is an input-conditioned restoration problem: the reconstructed HR image should be perceptually both realistic and faithful to the LR input. Early deep learning based SR methods 9 ; 59 ; 21 ; 58 ; 7 mainly optimize pixel-wise objectives such as \ell_{1} loss, which improve distortion measures but often produce over-smoothed textures. GAN-based methods 18 ; 39 ; 38 improve perceptual sharpness, but often suffer from unstable adversarial optimization and visible artifacts. More recently, diffusion models 15 ; 31 have emerged as a powerful alternative to GAN, combining stable training with strong generative capacity. Early vision-only diffusion SR methods, such as SR3 28 , SRDiff 19 and ResShift 53 , have demonstrated promising detail synthesis performance, but struggle in challenging real-world scenarios where the LR inputs show semantic ambiguity and complex degradations. A recent line of research addresses this limitation by adapting pre-trained text-to-image (T2I) diffusion models, such as Stable Diffusion 32 ; 25 ; 12 , for SR 37 ; 45 ; 34 ; 48 ; 22 ; 6 ; 11 . In general, these methods start with a generic T2I generator and then enforce compliance with the LR input through prompts, adapters, or other control mechanisms. Although effective, these methods adapt a generic T2I generator to process the LR input, rather than train a restoration model directly for detail generation. For SR, however, being perceptually realistic alone is insufficient; the restored details must also be faithful to the LR observation. Unfortunately, the multi-modal pre-training nature of T2I models introduces semantics through text or text-aligned representations 45 , increasing the risk of detail hallucination. Even when derived from the LR input itself, such cues are still injected through a text-conditioned generative path, which is often spatially coarse and weakly grounded. These observations motivate us to raise a fundamental question: can a purely vision-based generative model, trained directly for restoration without relying on multimodal pre-training, rival T2I-based SR models? In this work, we answer this question by proposing VOSR , a V ision- O nly generative model for image S uper- R esolution. Here, vision-only means that the model is trained only on visual data, including synthesized LR-HR pairs and auxiliary features from vision encoders, without multimodal text-image supervision. Rather than imitating the T2I training paradigm, VOSR follows a native restoration-oriented design. First, we introduce vision semantic condition , which extracts semantically rich features directly in the visual domain from the LR image using pretrained vision encoders such as DINO 24 ; 29 . Unlike text-aligned semantics, these features are tightly grounded to the input and better suited to resolving fine-grained structural and texture ambiguities. Second, we revisit classifier-free guidance (CFG) 16 for vision-only restoration problems. In modern diffusion models, CFG is a critical mechanism for boosting perceptual quality and generative sharpness, yet existing vision-only SR methods 28 ; 19 ; 53 mainly focus on conditional restoration itself and rarely study guidance as an explicit tool for improving restoration quality. At the same time, directly adopting the standard fully unconditional auxiliary branch is suboptimal for SR: once the LR condition is entirely removed, the auxiliary branch must learn generic generation from scratch, while the conditional branch is left to carry input controllability. This role split is difficult to optimize in restoration, and a poorly learned unconditional branch can undermine generation quality while providing a weaker reference for restoration-oriented guidance. We therefore propose a restoration-oriented guidance that replaces the unconditional branch in CFG with a partially conditioned branch. Specifically, weak LR structural cues are preserved while the high-level semantics are dropped. This keeps the auxiliary branch generative yet explicitly input-anchored, making the resulting guidance direction restoration-oriented. Consequently, VOSR strengthens input-consistent restoration rather than amplifying generic generative preference. Interestingly, it also induces a different inference behavior from conventional T2I-based guidance: larger guidance scales favor fidelity by moving predictions closer to the fully LR-conditioned branch, while smaller scales allow stronger generation by leaning toward the partially conditioned branch. Beyond restoration quality, inference efficiency is also critical for practical SR. We therefore first train a multi-step VOSR model and then distill it into a one-step variant for fast deployment. Fig. 1 compares VOSR with generative SR methods in terms of performance, efficiency, and training cost across both T2I-based and vision-only approaches under multi-step and one-step settings. As shown in the figure, VOSR is competitive with or better than most T2I-based SR methods in both regimes, while outperforming significantly previous vision-only methods in perceptual quality. Our multi-step model is substantially more efficient than existing T2I-based SR systems, and the one-step variant remains comparable to recent one-step T2I methods while enabling fast deployment. Moreover, VOSR requires only about one-tenth of the training cost of representative T2I-based SR methods. These results suggest that a restoration-oriented, vision-only framework can simultaneously offer strong perceptual quality, practical efficiency, and much lower training cost without multimodal pretraining. 2 Related Work Vision-Only Image Super-Resolution . Image super-resolution has traditionally been formulated as a purely visual restoration problem. Early deep models, including convolutional networks 9 ; 59 ; 60 and vision transformers 21 ; 58 ; 7 , are usually trained with pixel-wise objectives such as \ell_{1} loss, which achieve strong distortion-based measures ( e.g . , PSNR) but often produce over-smoothed textures. To improve perceptual quality, later methods incorporate perceptual losses 57 or adversarial training 39 ; 54 ; 38 , but these approaches often suffer from training instability and visible artifacts. Diffusion models 15 ; 31 have recently emerged as a powerful alternative for generative SR. Methods such as SR3 28 , SRDiff 19 , ResShift 53 , and SinSR 40 show that high-quality detail synthesis is possible without multimodal supervision. From a task perspective, this paradigm is naturally aligned with restoration because generation is driven directly by the degraded input. However, existing vision-only generative SR methods mainly condition restoration through structural cues from the LR input, without explicitly introducing semantic guidance. Such conditioning often provides insufficient high-level information under severe degradation. In other words, the limitation of vision-only SR does not lie in its task formulation but its lack of sufficiently strong semantic abstraction and restoration-oriented guidance. T2I-based Image Super-Resolution . The success of large-scale T2I diffusion models and related generative foundation models in a broad range of visual generation tasks 27 ; 25 ; 12 ; 56 ; 36 ; 46 has inspired a new line of SR methods that adapt pretrained T2I backbones for restoration 37 ; 45 ; 34 ; 48 ; 33 ; 26 ; 1 ; 6 ; 11 . These methods benefit from strong priors learned from massive image-text corpora and often achieve impressive perceptual quality. Typical designs inject LR information through prompts, ControlNet-like branches, adapters, or text-aligned features. Despite their success, T2I-based SR adopts a rather different training paradigm from native image restoration. Rather than directly training a restoration model, these methods constrain a generic image generator to comply with the LR image. While highly effective in leveraging pre-trained image priors, they introduce a structural tension between generic image generation and faithful restoration. Moreover, the semantic cues used by these methods are often represented in text or text-aligned spaces, which are spatially coarse to align with pixel-level image details. In addition, this paradigm inherits the high training and deployment cost of multimodal foundation models. Toward Restoration-Oriented Generative SR. The above two paradigms reveal an important issue in current generative SR research. On the one hand, vision-only SR is naturally grounded in the degraded input and better aligned with the restoration objective, yet its generative capability is limited by weak semantic abstraction under severe degradation. On the other hand, T2I-based SR provides powerful image priors, yet its generic multimodal generation framework is not native to restoration and is often constrained by the model size of the pretrained backbone, making lightweight, budget-aware restoration design challenging. These observations raise an inspiring question: can we train a generative SR model that remains fully grounded in the input LR image while incorporating stronger semantics directly in the visual domain? Meanwhile, recent one-step and few-step acceleration methods in generative modeling 30 ; 13 ; 51 ; 23 ; 14 , together with emerging efficient SR approaches 44 ; 33 ; 3 ; 10 ; 50 ; 52 , indicate that generative SR should ideally support both high perceptual quality and efficient inference. In this paper, we develop a vision-only generative SR framework along this line. 3 Method Figure 2 : Overview of VOSR. (a) Framework overview. Given an LR image, VOSR builds two complementary conditions from the input: a spatially aligned structural condition in the VAE latent space and a high-level visual semantic condition extracted by a pretrained vision encoder. These conditions are injected into a diffusion transformer to predict the denoising velocity for HR reconstruction. (b) Condition and guidance design. Compared with prior vision-only SR that mainly relies on structural-only conditioning, VOSR introduces an additional visual semantic condition to reduce semantic ambiguity in restoration. Moreover, instead of using a fully unconditional branch as in standard classifier-free guidance, VOSR adopts a restoration-oriented guidance and removes semantic guidance while retaining weakened LR structural cues, making the guidance input-anchored and more suitable for restoration. 3.1 Framework Overview Given a low-resolution image I_{\mathrm{LR}} , the goal of SR is to recover a high-resolution image I_{\mathrm{HR}} that is perceptually realistic and faithful to the input. Unlike text-to-image generation, SR is purely vision-conditioned. Based on this property, we build VOSR , a vision-only generative SR framework, whose overall pipeline is shown in Fig. 2 (a). Following latent diffusion 27 , we encode the target HR image into a latent code z_{0}=\mathcal{E}(I_{\mathrm{HR}}) with a VAE encoder \mathcal{E} , sample Gaussian noise z_{1}\sim\mathcal{N}(0,\mathbf{I}) , and define the intermediate latent at time t\in[0,1] as z_{t}=(1-t)z_{0}+tz_{1} . The diffusion backbone then predicts the velocity target v_{t}=z_{1}-z_{0} from z_{t} under two LR-derived conditions: low-level structural conditioning and high-level visual semantic guidance . Specifically, we encode the LR image into the same latent space as a spatially aligned structural condition to preserve content and layout, and extract semantic features from a pretrained vision encoder as visual semantic guidance to provide semantically rich and spatially grounded cues for detail generation. Fig. 2 (b) highlights the proposed condition and guidance strategies. For the condition design, prior vision-only SR mainly relies on structural-only conditioning 28 ; 53 . In contrast, VOSR further introduces visual semantic embedding. The two conditions are complementary: the structural condition preserves fidelity, while semantic guidance helps resolve ambiguity in HR reconstruction. For the guidance strategy during training, instead of removing all conditions in the unconditional branch in standard classifier-free guidance, we propose a partially conditioned branch, where we remove semantic guidance and weaken the LR structural guidance. This keeps the guidance anchored to the input and better suits restoration. Built on the above design, we first train a multi-step model for high-quality generation and then distill it into a one-step model for efficient inference. In this way, VOSR unifies vision semantic condition, restoration-oriented guidance, and one-step distillation in a single vision-only SR framework, as introduced in detail below. 3.2 Vision Semantic Condition We first specify the two conditioning strategies in Fig. 2 (b). Existing vision-only generative SR models 28 ; 53 mainly rely on structural cues from the LR input. However, under severe degradation, structural conditioning alone can be insufficient because the LR observation admits multiple plausible HR reconstructions, causing semantic ambiguity. VOSR therefore adopts a two-branch conditioning design that combines LR structure with semantic information. Specifically, the structural branch encodes the LR image with the same VAE encoder \mathcal{E} as used to map HR images into the latent space, producing a structural condition c_{\mathrm{str}}=\mathcal{E}(I_{\mathrm{LR}}) to preserve spatial content. In parallel, a pretrained vision encoder \mathcal{V} , such as DINO 24 ; 29 , extracts a semantic condition c_{\mathrm{sem}}=\mathcal{V}(I_{\mathrm{LR}}) . Unlike text or text-aligned representations, c_{\mathrm{sem}} is extracted entirely in the visual domain and is better suited to fine-grained restoration. Our denoising backbone is a diffusion transformer 49 that takes the noisy HR latent z_{t} and timestep embedding as input. The structural condition c_{\mathrm{str}} is injected as spatially aligned latent conditioning, while the semantic condition c_{\mathrm{sem}} is introduced through cross-attention to provide high-level context. In this way, structural features preserve spatial fidelity, while semantic features resolve ambiguity during detail synthesis. This design introduces high-level semantics entirely within the visual domain, avoiding text or text-aligned conditioning in SR. 3.3 Restoration-Oriented Guidance After defining the visual conditions, we discuss how to guide generation toward perceptually realistic yet input-faithful restoration. Classifier-free guidance (CFG) 16 is a key technique in modern conditional diffusion models, as it can substantially improve sample sharpness and controllability at inference time. However, despite its importance in conditional generation, CFG remains largely underexplored in existing vision-only SR methods 28 ; 19 ; 53 . We argue that, for restoration, the design of the auxiliary guidance branch is especially important, because SR is not text-conditioned generation but input-anchored generation. We examine the standard CFG for SR, as shown in Fig. 2 (b). During training, the model randomly switches between a fully conditioned branch and an unconditional branch. At inference, the guided prediction is computed as v_{\mathrm{cfg}}=v_{\mathrm{uncond}}+s\left(v_{\mathrm{cond}}-v_{\mathrm{uncond}}\right) , where v_{\mathrm{cond}} and v_{\mathrm{uncond}} denote the conditional and unconditional predictions, respectively, and s is the guidance scale. This formulation is highly effective in T2I generation, where the unconditional branch provides a natural reference for generic image generation without text control. For SR trained from scratch, however, such a fully unconditional branch is less suitable. Once the LR input is entirely removed, the auxiliary branch must learn generic image generation, while the conditional branch is solely responsible for maintaining fidelity to the input. This separation of roles makes it difficult to optimize for image restoration, and weak unconditional learning may provide a poor guidance reference. Based on this observation, we replace the unconditional branch with a partially conditioned branch. The goal is not to drop conditions arbitrarily, but to construct an auxiliary branch, which is different from the fully conditioned branch and defines a meaningful restoration-oriented guidance direction. Specifically, instead of removing all conditions, we retain weakened LR structural cues while dropping semantic guidance. Compared with the fully conditioned branch using both structural and semantic conditions (c_{\mathrm{str}},c_{\mathrm{sem}}) , the partially conditioned branch keeps only a scaled structural condition and removes semantic conditioning. Let \alpha\in(0,1) denote the structural retention factor. The two branches are defined as: v_{\mathrm{cond}}=v_{\theta}(z_{t},t,c_{\mathrm{str}},c_{\mathrm{sem}}) and v_{\mathrm{pcond}}=v_{\theta}(z_{t},t,\alpha c_{\mathrm{str}},\varnothing) , where \varnothing denotes the absence of semantic conditioning. In this way, both branches remain anchored to the LR input, while their difference captures the effect of stronger structure and semantic guidance. During training, we randomly sample either the fully conditioned mode or the partially conditioned mode so that the shared model learns both behaviors under a unified objective. Following standard latent diffusion training in velocity parameterization, the multi-step objective is \mathcal{L}_{\mathrm{ms}}=\mathbb{E}_{z_{0},z_{1},t,\kappa}\left[\left\|v_{\theta}(z_{t},t,\kappa)-v_{t}\right\|_{2}^{2}\right], (1) where v_{t}=z_{1}-z_{0} is the ground-truth velocity target, and \kappa denotes the sampled conditioning mode, i.e . , either the fully conditioned branch (c_{\mathrm{str}},c_{\mathrm{sem}}) or the partially conditioned branch (\alpha c_{\mathrm{str}},\varnothing) . At test time, we apply restoration-oriented guidance: v_{\mathrm{cfg}}=v_{\mathrm{pcond}}+s\left(v_{\mathrm{cond}}-v_{\mathrm{pcond}}\right), (2) where s is the guidance scale. Unlike standard CFG, the auxiliary branch here is still weakly conditioned on the LR input. Therefore, the guidance direction does not move from unconditional generation toward conditioned restoration; instead, it moves from a weakly anchored branch toward a strongly anchored one. As s increases, the prediction is driven closer to the fully conditioned branch, which strengthens input consistency. As s decreases, the prediction leans toward the partially conditioned branch, where semantic conditioning is removed and structural conditioning is weakened, thereby allowing a larger space of plausible detail generation. This behavior is notably different from the CFG commonly used in T2I-based SR 45 ; 6 , which can be written as: \displaystyle v_{\mathrm{cfg}}={} \displaystyle v_{\theta}(z_{t},t,c_{\mathrm{lr}},\varnothing) (3) \displaystyle+s\Big(v_{\theta}(z_{t},t,c_{\mathrm{lr}},c_{\mathrm{sem}})-v_{\theta}(z_{t},t,c_{\mathrm{lr}},\varnothing)\Big), where the LR control is shared by both branches. The direction of guidance mainly reflects the injection of additional semantic priors, and increasing the guidance scale usually amplifies generative semantics. In our formulation, by contrast, s=0 recovers the partially conditioned branch and s=1 the fully conditioned branch. Hence, a smaller s tends to produce more generative results, whereas a larger s favors better fidelity to the LR input. This makes our VOSR restoration-oriented, which balances generation and fidelity within an input-anchored restoration space. 3.4 One-Step Distillation for Efficient Inference Although the multi-step VOSR model provides strong restoration quality, iterative sampling is still expensive for practical deployment. We therefore distill it into a fast student while preserving the same vision semantic condition and restoration-oriented guidance as the teacher. That is, distillation changes only sampling efficiency, not the restoration formulation. We parameterize the student as f_{\theta}(z_{t},t,r,\kappa) , where the auxiliary target time r denotes the target timestep to be predicted by the student, unifying local prediction ( r=t ) and compressed cross-time prediction ( r