Video and Language Alignment in 2D Systems for 3D Multi-object Scenes with Multi-Information Derivative-Free Control
Jason Armitage, Rico Sennrich · 2025
Read Video and Language Alignment in 2D Systems for 3D Multi-object Scenes with Multi-Information Derivative-Free Control by Jason Armitage, Rico Sennrich (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-24826.txt.
Introduction
Jason Armitage and Rico Sennrich help two-dimensional vision-language systems work inside three-dimensional multi-object scenes by steering an in-scene camera. Models trained on flat images face a dimensional shift in 3D, and the camera must be learned as a control module. Their method improves multivariate mutual information estimates through regret minimization with derivative-free optimization.
Off-the-shelf cross-modal systems then adapt online to occlusions and tell features apart, learning from noisy vision-language outputs. Readers in embodied AI get a control scheme that needs no gradients through the perception model. Active viewpoint choice becomes a way to make flat-trained models useful in spatial scenes.
Catalog record
| CREATOR | Jason Armitage, Rico Sennrich |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 86 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-24826
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Video and Language Alignment in 2D Systems for 3D Multi-object Scenes with Multi-Information Derivative-Free Control Jason Armitage, Rico Sennrich Abstract Cross-modal systems trained on 2D visual inputs are presented with a dimensional shift when processing 3D scenes. An in-scene camera bridges the dimensionality gap but requires learning a control module. We introduce a new method that improves multivariate mutual information estimates by regret minimisation with derivative-free optimisation. Our algorithm enables off-the-shelf cross-modal systems trained on 2D visual inputs to adapt online to object occlusions and differentiate features. The pairing of expressive measures and value-based optimisation assists control of an in-scene camera to learn directly from the noisy outputs of vision-language models. The resulting pipeline improves performance in cross-modal tasks on multi-object 3D scenes without resorting to pretraining or finetuning. Vision-language models (VLMs) trained on 2D visual inputs ingest 3D scenes in the form of sequences of discrete viewpoints. Systems rely on this method to perform tasks where samples at test include an additional dimension Wang et al. 2022 ; Voigt et al. 2023 ; Xue et al. 2024 ; Liu et al. 2024 . Consider a visual sequence to be a set of views rendered from a camera moving over a 3D scene. In this case, the problem reduces to predicting in order the location of viewpoints over x -, y -, and z -axes. Given a system and a 3D reconstructed scene, we posit that an optimal sequence of 2D viewpoints exists, that is the sequence that - when paired with the respective linguistic representations - is most likely to return an accurate prediction from the VLM. Figure 1: Textures of objects in 3D scenes vary in appearance depending on the position of the in-scene camera. Aligning a description with a scene is even harder when referenced objects belong to a group such as a single boulder in an outcrop on Mars. An optimal sequence of viewpoints improves the decisions of VLMs trained on 2D data where understanding a 3D reconstructed scene relies on a set of views. Why should we be concerned with the precise locations of the viewpoints presented to the VLM? Reasoning over the features of several objects at the same time is highly challenging for these systems Linghu et al. 2024 . In the generation of 3D scenes, systems can expect with some confidence that the left door of a car will be the same colour as the right. The front view of a building is in contrast a less reliable indicator of the appearance of its roof or rear door. A high level of confidence in a VLM’s assessments is of utmost importance if these systems are to be of use for experts performing scientific analysis. In planetary science, features such as a specific boulder’s texture form the core evidence when performing geological analysis on visual data from a Mars rover (see Figure 1 ). Object appearances vary depending on the lighting at the time of day, wind conditions, and the position of the cameras in relation to the objects Bell III et al. 2022 ; Paar et al. 2023 . Rock surface details are highly variable: features of interest clearly visible on one side of a boulder may be covered by detritus on all other sides. In addition to these variances in 3D scenes, the internal dynamics of VLMs are also unknown. Generalisation is improved for tasks where the test samples are 2D by finetuning. In contrast, availability of cross-modal training data with 3D visual inputs is restricted Poole et al. 2023 . This problem is acute in exploratory science where visual evidence is novel and costly to generate. Additional training is compromised in frontier research by limitations on domain-relevant 3D assets Hofmann et al. 2023 ; Barnes et al. 2018 ; Yuan et al. 2024 . We propose a straightforward approach to predicting a VLM’s errors and tune an in-scene controller guided by a measure that quantifies both the information presented in the scene and model uncertainty. The success of this approach will rely on the ability of our measure to express the complexity of the scene. A measure of one or two features alone is insufficient - all available variables should be accounted for. The estimated value of this metric should combine the information for all variables in a way that limits redundancies - or regret as defined in Krichevsky 1968 - to a minimum. To limit compute and data costs, we target a process that avoids differentiable training associated with auxiliary models or finetuning the VLM. Information theory provides an efficient and principled framework in which to develop statistical approaches with provable bounds Kullback 1997 . We opt for a form of mutual information estimation over multiple variables (MI) as the basis for our metric. Assuming no access to the VLM’s parameters at inference time, we apply a zeroth-order algorithm (ZO) to optimise the expressiveness of our MI measure by reducing the redundancies between variables Chen et al. 2017 ; Malladi et al. 2023 ; Wollstadt et al. 2023 of the model’s predictions. Our multi-information metric Studenỳ 1987 optimised with ZO algorithm (MI-ZO) overcomes the theoretical challenges of estimating MI measures over n>2 Berrett et al. 2019 mixed discrete and continuous Gao et al. 2017 variables using active regret minimisation. Accurate and fast reasoning on 3D scenes is a key to unlocking the potential of integrating 2D VLMs into processes where automation complements human expertise. An acute challenge is presented by scenes populated with similar objects containing minor differences. In addition to forms of scientific analysis noted above, this challenge is evident in asset creation for 3D virtual environments. Examples include generating a set of buildings to populate a street and 3D representations of components in a particle accelerator Chen et al. 2022 ; Peixoto et al. 2020 ; Huang et al. 2022 . A deficit of efficient methods to evaluate and perform rapid testing on 3D generations slows development and constrains adoption of systems Hofmann et al. 2023 . New methods for assisting experts in planetary science and adjacent domains are also required as the volume of observations and data processing are on the rise National Aeronautics and Space Administration 2024 ; Cuillandre et al. 2024 . We formulate the problem of the dimensional shift presented by 3D scenes as adaptive control of the in-scene camera to return an optimal sequence of actions. In robotics, efficient methods from control theory are used to optimise the selection of sequences of views ( Swain and Stricker 1993 ; Tallamraju et al. 2019 ) . Here control of the camera is optimised by measuring the information of the visual scene in relation to the linguistic description. The result is a method for applications with complex 3D scenes that requires no access to the VLM’s parameters or costly backpropagation Malladi et al. 2023 . Our research presents four contributions to improve VLMs trained on 2D data in analysis performed on 3D multi-object scenes: • A new algorithm denoted as MI-ZO that estimates online multi-information with active regret minimisation on n>2 variables using zeroth-order optimisation. Our method reduces overlaps between a set of continuous and discrete variables representing cross-modal inputs with a novel application