Pay Attention to Where You Looked
Alex Berian*, JhihYang Wu*, Daniel Brignac, Natnael Daba, Abhijit Mahalanobis · 2026
Read Pay Attention to Where You Looked by Alex Berian*, JhihYang Wu*, Daniel Brignac, Natnael Daba, Abhijit Mahalanobis (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-18970.txt.
Introduction
Alex Berian and four co-authors improve photo making from just a few input views. Novel view synthesis builds a new photo of a scene from a fresh angle. With only a few shots to start from, older methods treat each input view as if it counts the same toward the target view.
That flat treatment leads to weaker results, since some views plainly help more than others. The team adds attention that weighs where the model looked, giving key views more say. Sharper few shot views help uses like product shots, scene previews, and robot sight where extra photos cost time.
Catalog record
| CREATOR | Alex Berian*, JhihYang Wu*, Daniel Brignac, Natnael Daba, Abhijit Mahalanobis |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-18970
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Pay Attention to Where You Looked Thanks: This work was supported by STR (under a contract with DARPA) and Arete (with STTR research funding). * Equal Contribution Code: https://github.com/alexberian/ViewAttention/ Alex Berian*, JhihYang Wu*, Daniel Brignac, Natnael Daba, Abhijit Mahalanobis Abstract Novel view synthesis (NVS) has advanced with generative modeling, enabling photorealistic image generation. In few-shot NVS, where only a few input views are available, existing methods often assume equal importance for all input views relative to the target, leading to suboptimal results. We address this limitation by introducing a camera-weighting mechanism that adjusts the importance of source views based on their relevance to the target. We propose two approaches: a deterministic weighting scheme leveraging geometric properties like Euclidean distance and angular differences, and a cross-attention-based learning scheme that optimizes view weighting. Additionally, models can be further trained with our camera-weighting scheme to refine their understanding of view relevance and enhance synthesis quality. This mechanism is adaptable and can be integrated into various NVS algorithms, improving their ability to synthesize high-quality novel views. Our results demonstrate that adaptive view weighting enhances accuracy and realism, offering a promising direction for improving NVS. † † address: University of Arizona, ECE Dept. Tucson, AZ, USA [berian, jhihyangwu, dbrignac, ndaba, amahalan]@arizona.edu 1 Introduction Novel view synthesis (NVS) has seen a surge of interest in this current age of generative modeling. While previous NVS method achieved significant results [ 9 ] , the introduction of diffusion models [ 13 , 11 ] has pushed the field of NVS closer to generating photorealistic images indistinguishable from real images. Due to diffusion’s generative power, many recent works [ 4 , 16 , 3 ] leverage a diffusion model’s capabilities when rendering the final image. In novel view synthesis, we are given a set of input images with respective camera poses and tasked with generating a new, unique target image from an unseen target camera pose. Neural radiance field (NeRF) style methods [ 9 , 1 , 2 ] use a dense set of input images to train a neural network to predict the target view’s color and density and proceed to volume rendering to obtain the final output image. More recent NVS methods assume we are given the target camera pose of the image we wish to generate [ 17 , 4 ] . The target camera pose is then used to cast rays in 3D space. These rays are then sampled to obtain latent vectors in the 3D world that can be transformed into each of the input image’s perspectives and used as input to a traditional NeRF model to obtain a rendered image. While the above methods make use of target camera information, they treat the importance of each source view relative to the target view as equal. Naturally, we would not expect this to be the case as some source views may contain little to no information pertaining to the queried target view. For example, if the target view queries the dorsal side of an object and we have 3 input views, 2 corresponding to frontal views and 1 corresponding to a side dorsal view, then we would expect the side dorsal input view to have a higher weighting when performing the rendering process. Figure 1 depicts this example. (a) Without weighting (b) With weighting Figure 1 : Example of the source camera weighting problem. (a) shows equal weighting of camera views even though some views capture non-useful information, and (b) shows prioritized weighting on the darker source camera as it is closest to the target view. In this work, we seek to solve this problem of source camera weighting with respect to the target view. We propose two schemes to accomplish this task: a deterministic weighting scheme and a cross-attention-based learning scheme. In the deterministic scheme, we experiment with various forms of deterministic weighting such as relative Euclidean distance from source views to the target view and error weighting based on differences in angles and camera distances from source views to the target view. In the learned scheme, we use the concept of cross-attention [ 15 ] for the model to learn which source views should attend to the target view more. Our experiments demonstrate that camera weighting outperforms the traditional scenario where all input poses are weighted equally. 2 Overview 2.1 Novel View Synthesis NVS is the problem of inferring a target image x_{t} from S source images \{x_{si}\}_{i=1}^{S} 1 1 1 For brevity, we denote \{x_{si}\}_{i=1}^{S}=\{x_{s1},x_{s2},x_{s3},\dots,x_{sS}\} . The target images and source images have associated camera pose matrices P_{t}\in\mathds{R}^{(4,4)} and \{P_{si}\}_{i=1}^{S} respectively. Few-shot NVS is when S is small (e.g. 5 or less). We seek to create a NVS model N_{\theta} with parameters \theta that predicts novel views \hat{x}_{t}=\ N_{\theta}(P_{t},\{x_{si}\}_{i=1}^{S},\{P_{si}\}_{i=1}^{S}). (1) 2.2 GeNVS GeNVS [ 4 ] encodes each source image into feature volumes W_{i}\in\mathds{R}^{128,128,64,16} (point clouds of 16-dimensional features) using a modified DeepLabV3+ [ 5 , 7 ] segmentation model. The volumes are oriented as truncated rectangular pyramids with respect to their source camera poses \{P_{si}\}_{i=1}^{S} and a dataset-specific z_{near} to z_{far} . A 16-channel feature image F\in\mathds{R}^{64,64,16} of the volumes is rendered via volume rendering from a target camera pose P_{t} . Stratified sampling is used to select points along each ray. A point r is trilinearly interpolated in each feature volume to get latent vectors \mathbf{l}_{i}(r)=W_{i}(r)\in\mathds{R}^{16} . The latent vectors are averaged together, then passed through a MLP f to get a 16-channel color c and density \sigma . c,\sigma=f\Bigg(\sum_{i=1}^{S}\mathbf{l}_{i}(r)\Bigg). (2) Once the color and density of every point along a ray are predicted, volume rendering is used to predict the 16-channel pixel value. The 16-channel feature image is used as conditioning to a denoising diffusion [ 8 ] U-Net U by concatenating it with 3-channel noise (or noisy target image). The denoising U-Net U outputs 3 channels. When sampling new images with multiple denoising steps, the feature image is skip-concatenated to the U-Net’s U input. The final output target image x_{t} is the output from the last denoising step. 2.3 PixelNeRF PixelNeRF, like GeNVS, predicts latent vectors \mathbf{l}_{i}(r) for points r along a ray for each source image x_{i} and uses a neural network to predict color c and density \sigma for volume rendering. PixelNeRF encodes each input image to a grid of latent vectors G_{i}\in\mathds{R}^{64,64,128} . A ray point r is projected onto S source image \{x_{si}\}_{i=1}^{S} plane then grid vector G_{i}(r)\in\mathds{R}^{128} is bilinearly interpolated. Each grid vector is concatenated with a positional encoding of the point’s r position in space, and the view direction with respect to the source pose P_{si} . This resulting vector is passed through a few fully connected ResNet [ 6 ] blocks to produce a latent vector \mathbf{l}_{i}(r) . As in equation ( 2 ) The latent vectors \{\mathbf{l}_{i}(r)\}_{i=1}^{S} are averaged followed by a few more fully connected ResNet blocks f to produce the final 3-channel color c and density \sigma of the point r . The rendered rays give the pixel values of the target image x_{t} . 3 Problem Statement PixelNeRF and GeNVS’s averaging of latent vectors does not take into account the relationship between the source poses \{P_{si}\}_{i=1}^{S} and the target pose P_{t} . ( 2 ) can