Modular Multimodal Classification Without Fine-Tuning: A Simple Compositional Approach
Herman Bergström, Aditya Mehrotra, Rahul G. Krishnan · 2026
Read Modular Multimodal Classification Without Fine-Tuning: A Simple Compositional Approach by Herman Bergström, Aditya Mehrotra, Rahul G. Krishnan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20674.txt.
Introduction
Herman Bergstrom, Aditya Mehrotra, and Rahul Krishnan present CoMET, a modular way to classify multimodal data without fine-tuning. Each modality passes through a frozen pretrained backbone, its codes are compressed with principal component analysis, and the results feed a tabular foundation model. Principal component analysis alone proves a strong adaptor across modalities in their tests. A lightweight adaptive pooler, PALPooling, further helps when foundation-model tokens align poorly with the task.
Catalog record
| CREATOR | Herman Bergström, Aditya Mehrotra, Rahul G. Krishnan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62939 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.20674
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Modular Multimodal Classification Without Fine-Tuning: A Simple Compositional Approach Herman Bergström, Aditya Mehrotra, Rahul G. Krishnan Abstract We introduce CoMET, C omposing M odality E ncoders with T abular foundation models , a simple yet highly competitive method for multimodal classification: pass each modality through a frozen pre-trained backbone, compress the resulting embeddings with PCA, and concatenate as input into a Tabular Foundation Model (TFM) for prediction. We show that PCA alone suffices to act as an adaptor yielding strong, robust performance across modalities. When the CLS tokens of the foundation model align poorly with downstream tasks, we propose PALPooling , a lightweight adaptive token pooler that consistently improves representation quality. By composing strong frozen representation learning backbones with TFMs, our approach achieves state-of-the-art results across diverse multimodal benchmarks without any training. On hierarchical tasks with large fine-grained class spaces, our approach enables fast and scalable classification, handling datasets with over 500,000 samples and 2,000 classes without any fine-tuning. Overall, our results show that the composition of foundation models is a simple, yet powerful, out-of-the-box solution for multimodal learning, challenging the necessity of complex, end-to-end training pipelines for new problems. 1 Introduction Tabular Foundation Models (TFMs) have recently demonstrated that strong predictive performance can be achieved by amortizing Bayesian inference via in-context learning ( Qu et al. 2025 ; Hollmann et al. 2022 ; Ma et al. 2024 ) . After pre-training, these models can be applied to new datasets without additional fine-tuning, avoiding the substantial training times that come with task-specific pipelines. Surprisingly, despite being trained on priors designed to mimic the data-generating process observed in tabular data, the models are effectively data-agnostic. Instead, we find that, by applying simple dimensionality reduction techniques, TFMs become competitive classification heads for representations extracted using modality-specific encoders, such as image and text (see section 5 ). This unexpected modality agnosticism raises a natural question: Can TFMs be used for fine-tuning-free multimodal classification? While there is limited initial work leveraging TFMs in multimodal setups, these either employ dataset-specific fine-tuning ( Kim et al. 2026 ) , or treat them as tabular encoders operating solely on tabular data ( Luo et al. 2025 ) . We argue that this not only detracts from the out-of-the-box nature of these models, but it is also not necessary to achieve competitive performance. Instead, we propose solving the task as a simple composition of foundation models, leveraging pre-trained encoders ( Siméoni et al. 2025 ; Clark et al. 2020 ) for non-tabular features, and a TFM to make the final predictions, a modular pipeline where components can be independently replaced or extended without retraining. Despite this simplicity, a potential bottleneck emerges in the interface between modality encoders and TFMs. In practice, representations are typically obtained via fixed pooling strategies, such as CLS -tokens or mean-pooling ( Dosovitskiy et al. 2020 ; Devlin et al. 2019 ) . When these embeddings fail to prioritize features relevant to the classification task, the final prediction quality may be limited, even when the backbone and TFM are individually strong. While adaptive pooling of individual tokens can traditionally help alleviate this issue ( Reimers and Gurevych 2019b ) , this would require fine-tuning, which in our setting would entail backpropagation through the TFM. To address this, we introduce Pseudo Attention Label Pooling (PALPooling). PALPooling is a lightweight pooling method that can be fit in seconds using one to three forward passes, preserving the training-free pipeline while enabling adaptive pooling in settings where backpropagation is not possible. In this work, we make the following contributions. i) We demonstrate that reducing the embedding dimensionality through PCA is enough to make TabICL a competitive multimodal classifier, without any further fine-tuning. ii) Given this finding, we propose CoMET , Composing Modality Encoders with Tabular foundation models , an easy-to-implement framework offering high classification accuracy with minimal implementation effort. iii) We introduce PALPooling, an adaptive pooling method that uses pseudo-attention labels to avoid backpropagation. The method demonstrates benefits for both image and text settings, outperforming CLS tokens that are misaligned with the target prediction, with minimal computational overhead. iv) We highlight that strong fine-tuning-free classification enables new applications, showcasing this in the setting of hierarchical prediction. 2 Related Work Tabular Foundation Models Prior-fitted networks were introduced by Müller et al. 2021 , and have since paved the way for a new approach to generalizable tabular foundation models ( Hollmann et al. 2022 ) . The typical approach, employed by models like TabPFN ( Hollmann et al. 2025 ) and TabICL ( Qu et al. 2025 ) , entails fitting the posterior predictive distribution on extensive synthetically generated data. However, models such as TabDPT ( Ma et al. 2024 ) show that competitive performance can be achieved when using large amounts of real-world data for the prior. While these TFMs were originally limited to small datasets with few features, recent iterations have begun to relax these constraints ( Qu et al. 2026 ; Grinsztajn et al. 2025 ) . Although the focus of this paper is to showcase the proof of concept, we anticipate that the underlying framework will naturally extend to other tasks such as causal effect estimation Balazadeh et al. 2025 . Multimodal Classification While there is extensive work exploring architectures that can efficiently integrate tabular features with other modalities, these often focus on a single non-tabular modality at a time. Bonnier 2024 propose TTT, a Tabular Text Transformer which constructs embedding vectors through a distance-to-quantile mechanism, but requires dataset-specific fine-tuning. TabSTAR ( Arazi et al. 2025 ) is closer to a pre-trained foundation model, but still requires a small amount of dataset-specific fine-tuning using Low Rank Adaptations ( Hu et al. 2022 ) . Another line of research utilizes contrastive learning between tabular and image features to learn joint representations, with models such as TIP ( Du et al. 2024 ) and MMCL ( Hager et al. 2023 ) . Still, these models require end-to-end fine-tuning on a per-dataset basis. Tabular Foundation Models on Multimodal Data TIME ( Luo et al. 2025 ) uses TabPFN to extract representations for the tabular features, and subsequently fuses these with image features (e.g., through concatenation) before training a linear classifier on top. Perhaps most similar to our setting, the recently proposed MMPFN ( Kim et al. 2026 ) , which forwards image and text features through the TabPFN backbone, but crucially employs fine-tuning with modality-specific projectors and a tabular alignment loss. As we show in Section 6.2 , this is not required, and we outperform their model using simple dimensionality reduction and TabICLv2. 3 Background Transformer-based foundation models have learned to encode rich structure across different data modalities ( Mur-Labadia et al. 2026 ; Siméoni et al. 2025 ; Clark et al. 2020 ) , outputting contextualized token embeddings. Concretely, given an input from modality m , the backbone \Phi_{m}(\cdot) produces a set of token embeddings corresponding to modality-specific units (e.g., image patches or subword tokens). To aggregate these tokens into a fixed-size representation suitable for downstream tasks, we typically leverage a pooling function, such as mean-pooling, learned attention-based pooling, or direct use of a dedicated classification ( CLS )-token, a special token trained to summarize the input according