LLM is Not All You Need: A Systematic Evaluation of ML vs. Foundation Models for text and image based Medical Classification

1 st Meet Raval, 2 nd Tejul Pandit*, 3 rd Dhvani Upadhyay · 2026

Read LLM is Not All You Need: A Systematic Evaluation of ML vs. Foundation Models for text and image based Medical Classification by 1 st Meet Raval, 2 nd Tejul Pandit*, 3 rd Dhvani Upadhyay (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-16549.txt.

Introduction

Meet Raval, Tejul Pandit and Dhvani Upadhyay weigh ML and base models for med sorts. Their 2026 preprint deals with sorts that join pics and text. New vision and language tools open fresh paths for such sorts. Past tests oft span few sets or one data type. A fair test must span both text and pic sets at once.

The team runs one strict test on four open med sets. Sets span both types and both two class and more class sorts. Joint scores show where each tool type leads or lags. That map guides teams to the right tool per task. The work is a shared bench for med sorts.

Catalog record

CREATOR1 st Meet Raval, 2 nd Tejul Pandit*, 3 rd Dhvani Upadhyay
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-16549

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLM is Not All You Need: A Systematic Evaluation of ML vs. Foundation Models for text and image based Medical Classification 1 st Meet Raval, 2 nd Tejul Pandit*, 3 rd Dhvani Upadhyay Abstract The combination of multimodal Vision-Language Models (VLMs) and Large Language Models (LLMs) opens up new possibilities for medical classification. This work offers a rigorous, unified benchmark by using four publicly available datasets covering text and image modalities (binary and multiclass complexity) that contrasts traditional Machine Learning (ML) with contemporary transformer-based techniques. We evaluated three model classes for each task: Classical ML (LR, LightGBM, ResNet-50), Prompt-Based LLMs/VLMs (Gemini 2.5), and Fine-Tuned PEFT Models (LoRA-adapted Gemma3 variants). All experiments used consistent data splits and aligned metrics. According to our results, traditional machine learning (ML) models set a high standard by consistently achieving the best overall performance across most medical categorization tasks. This was especially true for structured text-based datasets, where the classical models performed exceptionally well. In stark contrast, the LoRA-tuned Gemma variants consistently showed the worst performance across all text and image experiments, failing to generalize from the minimal fine-tuning provided. However, the zero-shot LLM/VLM pipelines (Gemini 2.5) had mixed results; they performed poorly on text-based tasks, but demonstrated competitive performance on the multiclass image task, matching the classical ResNet-50 baseline. These results demonstrate that in many medical categorization scenarios, established machine learning models continue to be the most reliable option. The experiment suggests that foundation models are not universally superior and that the effectiveness of Parameter-Efficient Fine-Tuning (PEFT) is highly dependent on the adaptation strategy, as minimal fine-tuning proved detrimental in this study. *Corresponding author I Introduction Advanced Artificial Intelligence (AI) is rapidly changing clinical procedures in the healthcare industry, especially in the areas of medical data processing[Multi-domain ABSA data generation] and diagnostic support. A paradigm change has been made possible by the development of Large Language Models (LLMs) and their multimodal extensions (VLMs), which allow scalable analysis of heterogeneous data, such as unstructured clinical text, medical pictures, and integrated reports. In the past, structured data, manual feature engineering, and well-known supervised machine-learning (ML) models like Logistic Regression (LR), Support Vector Machines (SVMs), or Gradient-Boosted Trees (GBTs) have been crucial for clinical classification tasks like diabetes prediction, mental-health disorder classification, or the detection of pathological findings in medical images. Although these classical methods offer robustness, computational efficiency, and crucial interpretability, they inherently struggle with the increasing volume of complex, high-dimensional, and unstructured or multimodal data generated in modern healthcare settings. I-A The Rise of Foundation Models in Medicine Transformer-based LLMs, such as the Gemini [ 1 ] and GPT [ 2 ] families, have shown hitherto unprecedented capabilities in knowledge retrieval, complicated reasoning, natural language comprehension, and data generation tasks [ 3 , 4 , 5 ] . LLMs are now being used in RAG systems, which are showing remarkable efficiency in clinical natural language processing (NLP) for a variety of applications, including clinical text generation, clinical note categorization, patient-report summarization, decision support, and medical education [ 6 , 7 ] . Simultaneously, Vision-Language Models (VLMs) [ 8 ] have started to simplify medical imaging analysis by fusing strong image encoders with language-based outputs, making tasks like content-based image retrieval, automated report production, and diagnostic reasoning easier [ 9 ] . These advances have the potential to significantly reduce the difficulty in implementing sophisticated categorization techniques in a wide range of medical inputs. I-B Limitations in Current Benchmarking Despite this quick development and clear promise, there are still a number of serious flaws in the research that prevent these foundation models from being widely and intelligently adopted in high-stakes clinical settings: • Absence of Cross-Modality Alignment: Previous studies have mostly focused on text-based or image-based classification separately. Unified experimental frameworks that perform simultaneous and controlled comparisons across both modalities (text and image) utilizing the same assessment metrics and methodologies are critically lacking. • Inconsistent Benchmarking Rigor: Evaluations frequently concentrate primarily on the superiority of one big model over another or compare only particular LLM paradigms (such as zero-shot prompting). The entire range of modeling techniques, classical ML pipelines, zero/few-shot prompt-based LLMs, and Parameter-Efficient Fine-Tuned (PEFT) LLMs [ 10 ] , are rarely thoroughly benchmarked side by side under similar dataset splits and pre-processing conditions [ 11 ] . • Underexplored Complexity: Compared to more straightforward binary tasks, multiclass classification—which is crucial for differential diagnosis, particularly in medical imaging with VLMs and refined multimodal models—remains much understudied. I-C Research Objective This work offers a thorough, four-task experimental comparison intended to provide empirical clarity on the performance landscape of contemporary foundation models vs well-established machine learning techniques in medical categorization in order to fill in these essential gaps. Four freely accessible datasets that cover binary and multiclass tasks in text and image modalities are used. We evaluated classical ML pipelines (logistic regression, LightGBM), prompt-based LLMs (Gemini 2.5 Flash for text, Gemini 2.5 multimodal for images [ 12 ] ), and fine-tuned LoRA-adapted [ 13 ] LLMs (Gemma3_Instruct_1B for text, Gemma3 + LoRA for vision [ 14 ] ). Our research aims to answer the following key questions: • RQ1: For binary and multiclass medical text classification problems, how do prompt-based and refined LLMs compare to traditional machine learning techniques? • RQ2: How do Vision-LLM pipelines and classical CNNs fare in medical image categorization, both in zero-shot and fine-tuned settings? • RQ3: When compared to standard ML in the medical field, under what particular circumstances (modality type, label complexity, dataset characteristics) do LLM-based techniques offer proven performance gains, or fail to do so? Our study offers a robust, apples-to-apples comparison by rigorously harmonizing pre-processing, data splitting techniques, and evaluation criteria across all workloads. For medical AI practitioners, choosing between tried-and-true ML techniques and the rapidly changing field of big transformer-based solutions for practical healthcare deployment, the ensuing insights are crucial. The paper is structured such that Section II offers Related Work, which reviews earlier studies on large-language models, vision-language models, and classical machine learning in the medical field. The datasets utilized are described in Section III , together with information on classification complexity (binary vs. multiclass) and modalities (text vs. image). Our methodology, including the specific design of every pipeline and model, is described in Section IV . The experimentation results, metrics used, and performance outcomes are reported in Section V . Section VI concludes with a discussion of future research directions. II Related Work The investigation of machine-learning (ML) and Large Language Model (LLM) methods for medical classification spans three primary research streams: (A) conventional ML for structured and unstructured medical data; (B) LLMs for medical text classification; and (C) multimodal and Vision-LLM (VLM) modelling in medical imaging. We reviewed each area and then outline how our work addresses the remaining critical gaps. II-A Conventional ML for Medical Classification Historically, many medical diagnostic classification tasks (e.g., diabetes prediction, skin-lesion classification) have relied on well-understood feature engineering and classical supervised ML pipelines, such as Logistic Regression (LR), Support Vector Machines (SVMs), and Gradient-Boosted Trees (GBTs). These approaches benefit from simplicity, interpretability, relatively low computational cost, and established validation protocols. For example, ML pipelines that use TF-IDF and structured features have shown robust performance in clinical-note classification settings [ 15

Read the complete text