Firebolt-VL: Efficient Vision-Language Understanding with Cross-Modality Modulation Quoc-Huy Trinh, Mustapha Abdullahi, Bo Zhao, Debesh Jha Abstract Recent advances in multimodal large language models (MLLMs) have enabled impressive progress in vision-language understanding, yet their high computational cost limits deployment in resource-constrained scenarios such as personal assistants, document understanding, and smart cameras. Most existing methods rely on Transformer-based cross-attention, whose quadratic complexity hinders efficiency. Moreover, small vision-language models often struggle to precisely capture fine-grained, task-relevant visual regions, leading to degraded performance on fine-grained reasoning tasks that limit their effectiveness in the real world. To address these issues, we introduce Firebolt-VL, an efficient vision-language model that replaces the Transformer-based decoder with a Liquid Foundation Model (LFM) decoder. To further enhance visual grounding, we propose a Token-Grid Correlation Module, which computes lightweight correlations between text tokens and image patches and modulates via the state-space model with FiLM conditioning. This enables the model to selectively emphasize visual regions relevant to the textual prompt while maintaining linear-time inference. Experimental results across multiple benchmarks demonstrate that Firebolt-VL achieves accurate, fine-grained understanding with significantly improved efficiency. Our model and code are available at: https://fireboltvl.github.io 1 Introduction Multimodal Large Language Models (MLLMs) have rapidly advanced vision-language capabilities, achieving strong performance on tasks such as image captioning 23 ; 27 , visual question answering (VQA) 2 ; 27 ; 47 , and optical character recognition (OCR) 41 ; 33 . Recent state-of-the-art models such as LLaVA 28 ; 26 ; 22 , IDEFICS 21 , OpenFlamingo v2 3 , MiniGPT-4 47 ; 5 , Chameleon 37 , InternVL 9 , Qwen-VL 4 , and FastV 6 underscore the growing importance of MLLMs in real-world applications. Nevertheless, deploying MLLMs remains challenging in resource-constrained settings due to their high computational and memory demands. Improving efficiency is therefore critical for enabling low-latency and on-device multimodal interaction, achieving seamless integration of linguistic and visual reasoning in practical technologies. Most recent MLLMs are built upon Transformer-based Large Language Models (LLMs), which exhibit quadratic computational complexity with respect to the input sequence length 39 ; 15 . As a result, inference is often inefficient on resource-constrained devices, and latency increases sharply for long-context multimodal inputs. Therefore, improving LLM efficiency is crucial to enable faster inference and facilitate the deployment of MLLMs in low-resource environments. (a) Task performance. (b) Throughput efficiency. Figure 1 : Performance and latency efficiency of Firebolt-VL on multiple tasks compared with state-of-the-art baselines. Firebolt-VL is both competitive and efficient, demonstrating strong generalization across diverse tasks. To alleviate the compute and memory overhead of MLLMs, several efficient architectures have been proposed, including Kosmos-2 34 , MobileVLM 42 , MobileVLM V2 10 , MoE-LLaVA 25 , LLaVA-Phi 48 , and SmolVLM2 31 . These models either leverage lightweight language backbones or incorporate Mixture-of-Experts (MoE) mechanisms 11 to reduce the number of active parameters and computational cost. Although such approaches have shown promising results on relatively simple benchmarks, such as image captioning and VQA, they still face two major challenges. First, small Transformer-based architectures exhibit quadratic computational complexity and limited capacity to model long-range dependencies. Second, there is a lack of precision when attending to task-relevant visual regions, which often leads to failures in handling fine-grained or detail-oriented questions that require rich visual representations. In this work, we address these limitations by introducing a token-grid correlation and modulation mechanism, implemented as the Cross-Modal Modulator (CMM). This module fuses visual grid representations with instruction text tokens to emphasize task-relevant visual cues. By doing so, the CMM enables the model to attend more effectively to fine-grained and detail-oriented information, enhancing its ability to reason over complex visual inputs. Furthermore, we explore the integration of the Liquid Foundation Model (LFM) 19 ; 1 with our proposed CMM for an efficient MLLM. From all of our proposed approaches, we design Firebolt-VL , an efficient MLLM for broad vision-language understanding evaluated on tasks including image captioning, VQA, chart understanding, and fine-grained visual reasoning, as shown in Figure 1 . In summary, our main contributions are fourfold: (1) We introduce Firebolt-VL , a novel MLLM that integrates the Liquid Foundation Model (LFM) 19 ; 1 for efficient sequence modeling, significantly reducing computational cost while maintaining strong multimodal reasoning performance. (2) We propose a Cross-Modal Modulator mechanism that fuses visual grid representations with text tokens. This enables more precise attention to task-relevant regions and improves the model’s capacity for fine-grained visual understanding. (3) We conduct extensive experiments on multiple benchmarks, including image captioning, VQA, and OCR. Results demonstrate that Firebolt-VL achieves competitive or superior performance compared to existing efficient MLLMs, while substantially improving inference efficiency and scalability. (4) We release the source code and pretrained model to promote transparency and encourage further research in the development of efficient MLLMs. 2 Related Work 2.1 Multimodal Large Language Model Multimodal large language models (MLLMs) have become a central research direction due to their wide applicability in document understanding, smart cameras, and virtual assistants. Recent advancements in state-of-the-art architectures 28 ; 21 ; 3 ; 47 ; 5 ; 26 ; 22 ; 37 ; 9 ; 4 ; 6 have achieved remarkable progress in visual understanding and text generation, bringing MLLMs closer to practical, real-world deployment. Despite these advancements, the computational demands of modern Vision–Language Models (VLMs) remain a major barrier. To deal with this challenge, early efforts such as MobileVLM 42 and MobileVLM V2 10 reduce the computational burden by employing lightweight Mobile-LLaMA backbones for text generation. Subsequent approaches, including MoE-LLaVA 25 and LLaVA-Phi 48 , adopt Mixture-of-Experts (MoE) 11 to activate only a fraction of parameters during inference, thereby eliminating redundant computation. More recently, SmolVLM2 31 introduced a language backbone combined with pixel-shuffle and inner-patching strategies to reduce the number of visual tokens to improve efficiency. While these models show promising performance and increasing adoption, they still rely heavily on Transformer-based architectures whose attention mechanism incurs quadratic time and memory complexity. This fundamental limitation restricts their ability to scale to long-context inputs and prevents truly lightweight, real-time deployment. To address this limitation, the design of Firebolt-VL integrates a Liquid-based 19 language model, which is leveraged by the Liquid Foundation Model (LFM) decoder 1 . This enables forward passes in linear-time complexity and significantly improves the overall efficiency of vision-language modeling. 2.2 Cross-modal Integration In recent works, most VLMs introduce cross-modal alignment through a simple linear projection layer, which maps visual features into a joint embedding space shared with the language encoder. While effective for large-scale models with strong visual backbones, this strategy becomes problematic for compact VLMs, whose vision encoders possess limited representational capacity, often resulting in weak or unstable alignment. To improve alignment quality in smaller models, several enhanced strategies have been proposed. Dense Connector 44 enriches the visual representation by aggregating multi-level features from earlier layers. Align-KD 12 leverages knowledge distillation to transfer cross-modal alignment cues from larger teacher models, thereby strengthening the alignment of compact VLMs. Building on this direction, Align-GPT 46 introduces a hierarchical alignment scheme that learns multiple alignment levels during pre-training and adaptively fuses them during instruction tuning to support diverse task requirements. Despite their effectiveness, these approaches still exhibit limited interactive fusion between visual and textual cues, often failing to direct the model’s attention toward the most relevant visual regions for a given instruction. Qwen-VL 4 addresses this issue by incorporating cross-attention between image and text tokens, enabling richer cross-modal interaction. However, cross-attention incurs quadratic computational complexity, making it unsuitable for lightweight or latency-constrained deployment. To overcome these limitations, in Firebolt-VL, we introduce the Cross-Modal Modulator (CMM), which is designed via the integration of a state-space model (SSM) 15 ; 17 ; 16 and FiLM 35 to fuse grid-level visual tokens with textual representations, and efficiently capture the long sequence features for the Large Language Model. By computing lightweight token–grid correlations and applying FiLM-based modulation within an SSM framework, CMM allows the model to dynamically emphasize the most informative visual elements while maintaining near-linear complexity. This design enables stronger fine-grained grounding and contextually accurate multimodal reasoning without the computational overhead of cross-attention, and it can be scaled further for dealing with the long video input or long sequence signal inference of the MLLMs. Figure 2 : Overview of the Firebolt-VL architecture. The Cross-Modal Modulator (CMM) fuses textual instructions with the visual representations of the query image to produce conditioned tokens, which are then processed by the Liquid Foundation Model (LFM). The model is trained in two stages: (1) CMM pre-training to initialize modulation parameters, and (2) end-to-end training of the full framework. 3 Method 3.1 Preliminaries State-Space Models (SSM). An SSM 18 is a sequence model that modifies the hidden state \mathbf{h}(t) over time through a linear dynamical system. Following this concept, the Structured State-Space Model (S4) 17 and Mamba 15 introduce a parameterization of \mathbf{A} that guarantees stability and expressiveness, allowing efficient training on long sequences while preserving global dependencies. S4 thus bridges the gap between the dynamical-system view of sequence modeling and the content-based attention mechanism 39 of Transformers. Feature-wise Linear Modulation (FiLM). Feature-wise Linear Modulation (FiLM) 35 is a lightweight yet effective conditioning mechanism that modulates one modality’s representation based on another by applying learned, feature-wise affine transformations. Given a visual feature vector \mathbf{x}\in\mathbb{R}^{D} and a conditioning signal \mathbf{c} (e.g., a text embedding), FiLM generates two modulation parameters, \gamma(\mathbf{c}) and \beta(\mathbf{c}) , through a learnable function such as a linear projection: \mathrm{FiLM}(\mathbf{x},\mathbf{c})=\gamma(\mathbf{c})\odot\mathbf{x}+\beta(\mathbf{c}), (1) where \odot denotes element-wise multiplication. Through this formulation, FiLM enables the conditioning signal to adaptively scale and shift visual features, integrating semantic cues into the representation without requiring explicit token-level attention. 3.2 Overview Figure 2 presents the overall architecture of our framework, which comprises three main modules: the Vision Encoder Vis(\cdot) , the Large Language Model LLM(\cdot) , and the proposed Cross-Modal Modulator CMM(\cdot) . 3.3 Vision Encoder For the vision encoder, we leverage the SigLIP 38 model, a multilingual vision–language encoder that replaces the traditional softmax contrastive objective with a sigmoid-based loss for image–text alignment. Given an input image X_{I}\in\mathbb{R}^{3\times H\times W} , the encoder first divides the image into several patches and processes them through a Vision Transformer (ViT) backbone to obtain global embeddings of grids X_{v}\in\mathbb{R}^{G\times D_{v}} , where G denotes the number of visual grids and D_{v} is the embedding dimension. In our framework, we employ the SigLIP encoder to extract grid-level visual embeddings X_{v} from the input image while preserving native aspect ratios. These embeddings are then projected into the shared latent space for multimodal fusion via the proposed Cross-Modal Modulator (CMM), ensuring fine-grained correspondence between textual cues and spatial visual features. 3.4 Large Language Model A key limitation of Transformer-based MLLMs lies in the quadratic computational complexity of the attention mechanism. Although FlashAttention has been proposed to reduce the complexity in the attention computation, it still poses a challenge due to the KV cache for calculating the attention weight. In this work, we replace the Transformer-based backbone with a Liquid hypothesis-based language model 19 , instantiated from the pretrained LFM2 text-only model 1 . This exploration improves MLLM efficiency as the Liquid-based model achieves substantially smaller time complexity than the attention-based model. Given the concatenated multimodal representation from CMM and the textual embeddings, denoted as H=[X_{t};X_{mm}]\in\mathbb{R}^{(L_{t}+L_{mm})\times D_{t}} , The model autoregressively generates the target sequence Y=\{y_{i}\}_{i=1}^{L_{y}} as depicted in Equation 2 . p_{\theta}(Y\,|\,H)=\prod_{i=1}^{L_{y}}p_{\theta_{i}}(y_{i}\,|\,H,y_{