LISA: Likelihood Score Alignment for Visual-condition Controllable Generation
Yanghao Wang Hongxu Chen Jiazhen Liu Zhenqi He, Rui Liu Zhen Wang Long Chen · 2026
Read LISA: Likelihood Score Alignment for Visual-condition Controllable Generation by Yanghao Wang Hongxu Chen Jiazhen Liu Zhenqi He, Rui Liu Zhen Wang Long Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27192.txt.
Introduction
Wang and colleagues improve how side networks steer image and video generators. In the dominant dual-branch design, a frozen main network makes pictures while a side branch encodes visual conditions like edges or poses. Score-based theory clarifies the split: the main branch supplies an unconditional score, a guide toward realistic images. The side branch quietly supplies a likelihood score, a guide toward matching the condition. Their LISA method makes that hidden role explicit in 2026 experiments.
Features are hooked from a chosen side-branch layer and projected by a small decoder into score space. An approximated likelihood score target is built, and the gap to that target becomes an extra training loss. Joint training with the normal diffusion loss speeds convergence markedly. Final image quality also improves across tasks, architectures, and model types. Side features grow more disentangled, easing conditional control. Extra inference cost is zero.
Catalog record
| CREATOR | Yanghao Wang Hongxu Chen Jiazhen Liu Zhenqi He, Rui Liu Zhen Wang Long Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.27192
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LISA: Likelihood Score Alignment for Visual-condition Controllable Generation Yanghao Wang Hongxu Chen Jiazhen Liu Zhenqi He, Rui Liu Zhen Wang Long Chen Abstract The prevalent dual-branch paradigm , i.e. , training a side network to encode visual conditions and fusing its intermediate-layer features to a frozen pretrained main network, has shown remarkable success in visual-condition controllable generation. Despite its widespread adoption, the role of the side branch and its training efficiency remain underexplored. In this paper, we first revisit this mainstream paradigm through the lens of score-based generative modeling: 1) The main network preserves visual perceptual quality by providing a prior unconditional score. 2) The side network steers conditional control by implicitly contributing a likelihood score. Guided by this perspective, we propose LIkelihood Score Alignment ( LISA ) , an effective regularization method that explicitly aligns the intermediate feature of the side network with an approximated likelihood score. Specifically, we first hook features from a designated layer of the side network and project them into the score latent space by a lightweight decoder. Then, we construct an approximated likelihood score target and calculate the distance between the decoder’s output and this target as an additional regularization loss. Finally, we jointly optimize the side network and decoder with both standard diffusion loss and our regularization loss. Experiments across various image/video tasks, architectures, and diffusion/flow models demonstrated that LISA can not only consistently accelerate the training convergence and improve final synthetic results, but also encourage the side network’s features to be more disentangled for conditional modeling with negligible additional training cost and zero extra inference cost. Figure 1: Likelihood score alignment (LISA) can improve training convergence and synthetic quality . Our framework, LISA, explicitly decomposes roles within the dual-branch paradigm: the main network and side network are responsible for the unconditional and likelihood score , respectively. By aligning a certain feature of the side network with an approximated likelihood score via a lightweight decoder, LISA can achieve >2.78\times faster convergence ( e.g. , as in ControlNet). 1 Introduction Recent advances in diffusion ( 11 ; 35 ) and flow matching models ( 21 ; 20 ) show remarkable visual generation capability. In particular, unconditional and text-conditioned generation tasks have been addressed quite well by billion-parameter models ( 7 ; 16 ; 38 ; 9 ) trained on large-scale and easy-to-collect training data. However, the increasing application requirements introduce a more challenging scenario ( 1 ) : Visual-condition Controllable Generation , i.e. , integrating visual-modality conditions, especially spatial conditions ( e.g. , pose, segmentation, and depth maps) for more fine-grained, structurally controllable image and video generation. To achieve this goal, prior studies ( 46 ; 24 ; 48 ) resort to a dual-branch paradigm ( c.f. , Figure 1 ): 1) Freezing a pretrained diffusion (or flow matching) model as the main network backbone. 2) Training a condition encoder as the side network to adopt the condition input and output a set of intermediate features. 3) Integrating these features into the main network’s original forward process to achieve conditional control. Under this paradigm, the follow-up study ( 41 ) extends representation alignment technology ( 43 ) to controllable generation, i.e. , aligning model features with a pretrained semantic encoder as an additional regularization to improve training efficiency. However, the dependence on external encoders bounds their performance to the chosen encoder. In this paper, we revisit this paradigm through the lens of score-based generative modeling: each branch network implicitly plays decomposed roles, and the feature-level integration mechanism essentially tries to induce an augmented result: 1) The frozen main network does not adopt condition input, i.e. , it is mainly responsible for providing an unconditional score \nabla_{\bm{x}_{t}}\log p_{t}(\bm{x}_{t}) to guarantee general perceptual quality. 2) The trainable side network encodes the condition \bm{c} and learns to bridge the gap between the conditional score \nabla_{\bm{x}_{t}}\log p_{t}(\bm{x}_{t}|\bm{c}) and the unconditional score \nabla_{\bm{x}_{t}}\log p_{t}(\bm{x}_{t}) , i.e. , achieving conditional control. 3) According to Bayes’ rule, this residue gap is the likelihood score , i.e. , \nabla_{\bm{x}_{t}}\log p_{t}(\bm{c}|\bm{x}_{t})=\nabla_{\bm{x}_{t}}\log p_{t}(\bm{x}_{t}|\bm{c})-\nabla_{\bm{x}_{t}}\log p_{t}(\bm{x}_{t}) . Our approach. Based on the above analyses, we argue that the main challenge in the dual-branch paradigm lies in training the side network, which learns to provide the control signal and implicitly corresponds to a likelihood score. Motivated by this, we propose LI kelihood S core A lignment (LISA), an effective regularization technique that explicitly aligns the side network with an approximated likelihood score. Since the frozen main network is naturally an unconditional score predictor and the paired training data can provide the closed-form conditional score, we can estimate a likelihood score by calculating the difference between them. By aligning the intermediate features of the side network with this approximated score alongside standard generative training, LISA introduces an efficient prior supervision. This explicit constraint acts as a regularization loss, significantly accelerating convergence and improving overall synthesis performance ( c.f. , Figure 1 ). Meanwhile, such regularization can encourage the side network’s features to be more disentangled for conditional modeling, thereby naturally demonstrating better compositional control. It is worth noting that, compared with existing representation alignment, our LISA does not require external semantic encoders, and achieves comparable improvements in training convergence and synthesis quality. Specifically, during the standard generative training, we first hook features from a designated layer of the side network and feed them into a lightweight trainable decoder (usually around 0.1\% size of the side network). This decoder only contains a few layers ( e.g. , convolution, activation, and upsampling layers) to transform the intermediate feature of the side network into a latent score space. Then, we calculate the distance between the decoder output and the likelihood score as a regularization loss, which is added to the diffusion loss as the final optimization objective. Notably, the additional training cost introduced by such regularization is almost negligible. During inference, we directly drop the decoder and use the trained side network for final conditional generation. We evaluated the effectiveness of our LISA across various image and video conditions, including pose maps, depth maps, low-resolution images, segmentation maps, and pose videos. All experimental results have consistently indicated that LISA can significantly accelerate the training convergence and bootstrap to a better synthetic quality (both perceptual quality and condition fidelity). Further evaluations, such as architecture-agnostic generalization and more challenging compositional controls, have verified that LISA is an effective and extensible solution for visual-condition controllable generation. In summary, our main contributions are as follows: • We analyzed the mainstream dual-branch paradigm for visual-condition generation from a novel perspective, revealing that the side network lacks explicit regularization for its intended role. • Based on the roles decomposition, we proposed an effective likelihood alignment method, LISA, which regularizes the side network’s intermediate output with an approximated likelihood score. • Extensive experiments across diffusion models, network architectures, and tasks have demonstrated significant and consistent gains on both training convergence and synthesis quality. 2 PRELIMINARIES We present a brief overview of diffusion and flow matching models via the unified perspective of stochastic process ( 15 ) and score matching ( 34 ) . The diffusion/flow models aim to capture the target data distribution p_{0} by learning a transport process from a prior distribution p_{T} ( e.g. , a