Batch Augmentation with Unimodal Fine-tuning for Multimodal Fusion of Large Language Models

H M Dipu Kabir, Subrota Kumar Mondal, Mohammad Ali Moni · 2025

Read Batch Augmentation with Unimodal Fine-tuning for Multimodal Fusion of Large Language Models by H M Dipu Kabir, Subrota Kumar Mondal, Mohammad Ali Moni (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06592.txt.

Description

H M Dipu Kabir and co-authors propose a 2025 training method for multimodal large language models. The method starts from pretrained models that each handle one kind of input. Each model is first fine-tuned alone, then joined through a Multi-Layer Perceptron head.

Training then uses batch augmentation across the combined model and its parts. Some parts can be swapped for scripts or AI agents when the task allows it. Tests used the FPU23 ultrasound set and the UPMC Food-101 set. The combined model gave the best average result among the methods tried. A shared dataloader script is also provided so others can repeat the training steps.

Catalog record

CREATORH M Dipu Kabir, Subrota Kumar Mondal, Mohammad Ali Moni
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-06592

Browse the full library · About the Tower of Babel