ByDeWay: Boost Your multimodal LLM with DEpth prompting in a training-free Way

Rajarshi Roy, Devleena Das, Ankesh Banerjee, Arjya Bhattacharjee, Kousik Dasgupta, Subarna Tripathi · 2025

Read ByDeWay: Boost Your multimodal LLM with DEpth prompting in a training-free Way by Rajarshi Roy, Devleena Das, Ankesh Banerjee, Arjya Bhattacharjee, Kousik Dasgupta, Subarna Tripathi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08679.txt.

Introduction

Rajarshi Roy and co-authors present a prompting trick that gives vision-language models a sense of depth, with no retraining, in a 2025 paper. Such models often fumble spatial questions and invent objects that are not there. Their method, ByDeWay, first estimates depth from the single photo, then slices the scene into near, middle, and far layers. A grounded vision model writes a short caption for each layer.

Those depth-tagged captions are pasted into the question prompt, so the main model reads the scene in layers. The trick is light, modular, and works even with closed models whose weights cannot be touched. On a test built to catch invented objects and on a hard reasoning test, scores rose across several models. The authors pitch depth as free context that any multimodal system can borrow.

Catalog record

CREATORRajarshi Roy, Devleena Das, Ankesh Banerjee, Arjya Bhattacharjee, Kousik Dasgupta, Subarna Tripathi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-08679

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ByDeWay: Boost Your multimodal LLM with DEpth prompting in a training-free Way Rajarshi Roy, Devleena Das, Ankesh Banerjee, Arjya Bhattacharjee, Kousik Dasgupta, Subarna Tripathi Abstract We introduce a training-free framework, ByDeWay , a training-free method to boost the performances of Multimodal Large Language Models. Specifically, ByDeWay leverages a novel prompting strategy, Layered-Depth-Based Prompting (LDP), that enhances the spatial reasoning and grounding capabilities of Multimodal Large Language Models (MLLMs). Our key insight is to inject structured spatial context derived from monocular depth estimation into the input prompts—without modifying any model parameters. By segmenting scenes into closest, mid-range, and farthest depth layers and generating region-specific captions using a grounded vision-language model, we produce explicit depth-aware textual descriptions. These descriptions are concatenated with image-question prompts to guide the model toward spatially grounded and hallucination-resistant outputs. Our method is lightweight, modular, and compatible with any black-box MLLM. Evaluations on hallucination-sensitive (POPE) and reasoning-intensive (GQA) tasks show consistent improvements across multiple MLLMs, demonstrating the effectiveness of depth-aware prompting in a zero-training setup. Figure 1 : ByDeWay Workflow. The light blue shaded area demonstrates the ByDeWay framework which utilizes Layered Depth-based Prompting (LDP) instead of standard prompting aiming to improve MLLM performances in a training-free way. The input image is processed to generate a depth map, which is then used to segregate visual information into layered regions (foreground, mid-ground, background). LDP augments standard prompting with the descriptions of such regions. MLLMs take input image, instruction, question and this augmented prompt as input and produces the response, the answer to the input question. Please note that the standard prompt goes to the input, shown as a dotted line, to the MLLMs for our baseline comparisons. 1 Introduction Multimodal Large Language Models (MLLMs) have emerged as powerful general-purpose systems capable of handling diverse tasks that span both language and vision modalities. From visual question answering (VQA) to image captioning and scene understanding, these models exhibit strong performance in zero-shot and few-shot settings by leveraging large-scale pretrained representations. Powerful MLLMs are unleashing new application potentials for digital marketing [ 14 ] and advertisements [ 17 ] . However, persistent challenges remain, chiefly hallucination —the tendency of MLLMs to generate content that is not grounded in the visual input—and reasoning‑sensitivity , where small changes in prompts or visuals can dramatically affect output. These issues become especially problematic in domains requiring precise spatial reasoning and object‑level grounding, such as robotics, healthcare, and safety‑critical applications. A recent survey finds that visual hallucinations—like misidentifying colors, objects, or spatial relations—continue to impede reliable deployment of MLLMs in real‑world settings [ 4 ] . To address the above challenges, we present ByDeWay ( B oost Y our multimodal LLM with DE pth prompting in a training-free Way ), a training-free method to boost the performances of Multimodal Large Language Models. Rather than fine-tuning the model or altering its architecture, we propose a lightweight and modular prompting strategy: LDP (Layered-Depth-based Prompting). LDP explicitly incorporates depth‑aware textual context into the prompt by describing discrete spatial layers of the scene—foreground, mid‑range, and background—based on captions generated from monocular depth maps. These layer‑specific descriptions act as structured, interpretable anchors, capturing pseudo‑3D scene organization without any model retraining. Concretely, we use an off‑the‑shelf monocular depth estimator to segment input images into depth‑based regions. Each region is captioned using a grounded vision–language model, yielding region‑specific textual summaries. These summaries are concatenated with the original image–question pair to form a depth‑enriched prompt , providing the MLLM with a clearer, spatially organized scene representation—thereby reducing hallucinations and bolstering visual reasoning in a fully training‑free manner (see Figure 1 ). Our key contributions are: • We introduce ByDeWay ( B oost Y our multimodal LLM with DE pth prompting in a training-free Way ), a framework for boosting multimodal large language models, using the below prompting strategy. • We propose Layered-Depth-based Prompting (LDP) , a novel prompting strategy that enriches MLLMs with explicit 3D spatial context using depth-aware captions. • Our method is modular and training-free , requiring no fine-tuning of the base MLLM or additional supervision, making it scalable and model-agnostic. • We demonstrate that LDP consistently improves performance on both hallucination-sensitive (POPE) and reasoning-intensive (GQA) benchmarks across multiple MLLMs. 2 Related Work 2.1 Multimodal Prompting Techniques Similar to Large language models (LLM), reasoning capabilities of large multimodal models are improved with queries designed for targeted applications. Prompting in MLLMs has largely been around designing the textual queries. Researchers have focused on methods like CoT prompting [ 20 ] , visual-context enrichment, and instruction tuning [ 25 , 18 ] . Models like BLIP-2 [ 8 ] and Flamingo [ 2 ] use both image and text inputs to adapt their prompts. While BLIP-2 utilize soft visual prompting , Flamingo performs interleaved prompt generation. However, multimodal prompting, where the prompting modality is different than the image and the text, is still largely unexplored in academia, partially because most of the recent open-sourced models are of limited capacity. A thread of studies explore having MLLMs predicting object locations as bounding boxes [ 19 , 10 ] or masks [ 11 ] . In recent work, set-of-marks [ 22 , 23 ] as a visual prompting notably improved the visual grounding abilities in the strongest MLLM. We introduce the concept of utilizing depth-based segmentation to enhance the prompt with spatial information. 2.2 Depth-Aware Caption Generation and Hybrid Multimodal Systems Traditional image captioning models such as “Show and Tell” [ 15 ] and bottom-up attention [ 3 ] primarily rely on 2D visual features, overlooking depth and spatial reasoning. Recently, several works have demonstrated that incorporating depth information can significantly enhance spatial understanding in captions. For example, DEVICE [ 21 ] enriches the TextCaps task by integrating depth maps and visual-concept tokens via a depth-enhanced feature updating module, leading to more accurate relations between scene objects. Similarly, Ahmed et al. [ 1 ] propose a Transformer-based RGB–D captioning framework that explores pixel-, feature-, and hybrid-level fusion strategies on the NYU‑v2 dataset, showing improved multi-sentence descriptions by leveraging depth cues. In parallel, dense captioning approaches operating on 3D point clouds go further by explicitly segmenting objects and reasoning about their inter-object relations. Scan2Cap introduced context-aware scene-object graphs and relational attention for dense descriptions of RGB–D scans [ 5 ] , while SpaCap3D [ 16 ] improved on this by introducing a token-to-token spatiality-guided Transformer encoder that learns fine-grained 3D spatial relations among objects. X‑Trans2Cap [ 24 ] further extends these ideas via cross-modal knowledge transfer, distilling RGB-aware features into a point-cloud-only model for efficient inference. Complementing these captioning systems, numerous segmentation architectures demonstrate that depth and RGB fusion enhance scene decomposition. For instance, HybridNet performs joint depth estimation and semantic segmentation using shared backbone branches [ 12 ] , and Fooladgar & Kasaei [ 6 ] adopt attentional fusion of RGB–D for improved segmentation performance. Moreover, modular multimodal pipelines like Descriptive Caption Enhancement (DCE) integrate specialized depth, spatial, and interaction modules to generate richer and more structured captions [ 13 ] . Our work draws on these strands by explicitly segmenting scenes according to depth and seamlessly integrating a depth-informed segmentation module into a hybrid captioning pipeline. In contrast to prior 2D-only or loosely coupled RGB–D models,

Read the complete text