From Images to Words: Efficient Cross-Modal Knowledge Distillation to Language Models from Black-box Teachers

Ayan Sengupta [email protected], Shantanu Dixit [email protected], Md Shad Akhtar [email protected], Tanmoy Chakraborty [email protected] · 2026

Read From Images to Words: Efficient Cross-Modal Knowledge Distillation to Language Models from Black-box Teachers by Ayan Sengupta [email protected], Shantanu Dixit [email protected], Md Shad Akhtar [email protected], Tanmoy Chakraborty [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10877.txt.

Introduction

Ayan Sengupta and coauthors present ARMADA. This framework distills knowledge from large vision-language models into smaller language-only models. It can use closed black-box teachers too. Standard distillation assumes teacher and student share a modality. Older multimodal routes need costly teacher pre-training. ARMADA instead aligns the models without changing the teacher. It was tested across twenty-five tasks. Those tasks span understanding, reasoning, and instruction tuning. It lifts student scores. It also challenges the idea that a vision teacher cannot strengthen a text model.

Catalog record

CREATORAyan Sengupta [email protected], Shantanu Dixit [email protected], Md Shad Akhtar [email protected], Tanmoy Chakraborty [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE86480
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.10877

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Images to Words: Efficient Cross-Modal Knowledge Distillation to Language Models from Black-box Teachers Ayan Sengupta [email protected], Shantanu Dixit [email protected], Md Shad Akhtar [email protected], Tanmoy Chakraborty [email protected] Abstract Knowledge distillation (KD) methods are pivotal in compressing large pre-trained language models into smaller models, ensuring computational efficiency without significantly dropping performance. Traditional KD techniques assume homogeneity in modalities between the teacher (source) and the student (target) models. On the other hand, existing multimodal knowledge distillation methods require modality-specific pre-training of the teacher model, which is computationally infeasible in most cases. In this paper, we introduce ARMADA , an efficient cross-modal knowledge distillation framework designed to transfer knowledge from large vision-language models, including black-box models, to language-only models. Unlike existing KD techniques that rely on the internal structures of multimodal teachers or require computationally expensive pre-training, ARMADA leverages novel alignment techniques to distil knowledge without altering the teacher model, ensuring efficiency and scalability. We empirically validate ARMADA on twelve natural language understanding, eight complex generative reasoning and five instruction-tuning tasks, demonstrating consistent performance improvements in large models such as DeBERTa-v2-1.4B, OPT-1.3B, LLaMA-{3B, 7B, 8B}. ARMADA achieves up to 3.4 % 3.4\% improvement on language understanding tasks and 2.6 % 2.6\% boost in generative reasoning, all without requiring expensive multimodal pre-training or fine-tuning of the teacher model. Our findings challenge conventional knowledge distillation paradigms by demonstrating that even vision-language models, despite lacking direct textual understanding, can significantly enhance language models when distilled appropriately. 1 Introduction Pre-trained large language models (LLMs) ( 1 ; 28 ; 41 ; 23 ; 19 ) have demonstrated remarkable performance across various tasks in diverse domains, including commonsense and mathematical reasoning, natural language understanding, code generation and dialogue generation, showcasing their versatility and effectiveness. However, as these models continue to grow in size and complexity, their computational costs become prohibitive for many applications. This has led to a surge of interest in model compression techniques ( 9 ; 34 ; 93 ) , which aim to reduce the size of a pre-trained LLMs for computational efficiency at test-time. 34 popularized the concept of knowledge distillation (KD) for compressing large models into smaller and shallower models. The majority of the subsequently proposed KD methods ( 73 ; 91 ; 93 ) are unimodal , i.e. , predominantly work when both the teacher and student models are from the same modality. The reliance on the teacher’s modality on student models prevents these methods from effectively transferring cross-modal cues and, therefore, restricts the student models from gaining a more generalized aspects of the downstream tasks. Cross-modal knowledge distillation has emerged as a technique where teacher models specializing in one modality can transfer knowledge to student models in another modality. This allows the student model to assimilate concepts across different modalities, analogous to how a prompter helps a blind person perceive the world through narration. Notable cross-modal KD approaches include ‘vokenization’ ( 74 ) , which associates text with visual tokens to improve language models, and video-language KD frameworks that utilize video-based teacher models for cross-modal knowledge transfer. In a similar attempt, 75 proposed a video-language KD framework, which leverages a video-language encoder teacher model for distilling cross-modal knowledge into the student LMs. However, the effectiveness of these existing cross-modal distillation techniques relies on the large pre-training of the multimodal teacher (e.g., 75 used 136M video clips for teacher pre-training). These computational expenses can be significantly reduced by utilizing readily available large pre-trained multimodal white-box ( 64 ) and black-box ( 7 ) models, which the existing cross-modal models fail to reap off. To this end, we propose ARMADA 3 3 3 The source code of ARMADA will be released upon acceptance of the paper. , a cross-modal knowledge distillation framework designed to distil knowledge from any white-box or black-box text-to-vision teacher model to language-based student models. At the core of ARMADA is the TS Aligner , a module that aligns the student model with the teacher’s multimodal abstraction space. ARMADA further utilizes novel manifold and auxiliary alignment mechanisms, which regularize students to learn different abstractions by jointly training the models with shared objectives. Unlike the existing cross-modal KD frameworks, ARMADA does not require student models to generate mental images but rather encourages abstract knowledge representation. This allows the framework to distil knowledge from a teacher model to different student models while maintaining efficiency and adaptability. We empirically validate ARMADA on twelve natural language understanding (NLU) tasks from the GLUE ( 79 ) and SuperGLUE ( 78 ) benchmarks, as well as five commonsense and three mathematical reasoning tasks. On NLU tasks, a BERT-6L model ( 21 ) distilled via ARMADA using Stable Diffusion ( 65 ) and Midjourney ( 8 ) teacher models achieves performance improvements of 3.4\% and 3.2\% , respectively, over its undistilled counterpart. ARMADA also enhances larger models (>1B parameters) such as DeBERTa-v2-xxlarge ( 33 ) ) and OPT-1.3B ( 89 ) , by 1.4\% and 1.5\% , respectively. Even in zero-shot generative tasks, ARMADA demonstrates its efficacy by improving the performance of a pre-trained LLaMA-7B model ( 77 ) by 0.5\% , with a maximum task-specific improvement of 2.6\% , proving its scalability across models of different sizes. Even with only 0.8\% learnable parameters than that of the existing unimodal and multimodal KD methods, ARMADA is proven to equally effective in assimilating teacher supervision into student models. The contributions of our paper are summarized below: • Cross-Modal Knowledge Distillation for Large Language Models. Unlike multimodal pre-trained models such as CLIP ( 61 ) or Qwen-VL ( 81 ) , in cross-modal distillation, the trainable model can access only one modality, making knowledge fusion immensely challenging. To our knowledge, ARMADA is the first architecture-agnostic cross-modal knowledge distillation technique from black-box teachers to language-only student models. • Efficient and Scalable Cross-Modal KD. Unlike existing cross-modal KD methods, our proposed method does not require computationally expensive modality-specific pre-training of teacher models. The framework enhances student models with minimal additional learnable parameters (0.8%) than the existing methods. This efficiency, combined with its ability to work with white-box and black-box teachers, positions ARMADA as a practical and scalable solution for improving language models with cross-modal signals. • Theoretical Insights into Cross-Modal Knowledge Distillation. We provide an analytical explanation of how cross-modal knowledge distillation works by establishing the equivalence between teacher and student manifold spaces. Although the subject being substantially least understood, our work theoretically justifies the effectiveness of cross-modal distillation for transferring abstract knowledge across modalities. This deepens our understanding of how knowledge transfer can be achieved across modalities, making the process both explainable and efficient. 2 Related Work Unimodal Knowledge Distillation. 34 expanded the work of 9 to introduce and popularize KD in which a shallow model tries to mimic the performance of a complex and large model through its output distribution. Instead of guiding the student through only the output layer of the teacher model, learning from intermediate features of the teacher has been proposed for better knowledge transfer ( 66 ; 86 ; 73 ; 88 ) . Other studies on distillation encouraged different techniques, including relation-based KD ( 59 ; 12 ) and meta KD ( 57 ; 93 ; 69 ) , for effective knowledge sharing. Cross-modal

Read the complete text