Large Language Models for Biomedical Article Classification
Jakub Proboszcz, Paweł Cichosz · 2026
Read Large Language Models for Biomedical Article Classification by Jakub Proboszcz, Paweł Cichosz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11780.txt.
Introduction
Proboszcz and Cichosz test large language models as classifiers of biomedical articles, across many setups. They vary prompts, output handling, and few-shot example choices, for both open and closed models. The best setups reach average PR AUC above 0.4 with zero-shot prompting, and higher with few-shot prompting. That sits close to naive Bayes, random forest, and fine-tuned transformer scores. Using output token probabilities for class probabilities is one setup they single out as promising.
Catalog record
| CREATOR | Jakub Proboszcz, Paweł Cichosz |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 105829 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.11780
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Large Language Models for Biomedical Article Classification Jakub Proboszcz, Paweł Cichosz Abstract This work presents a systematic and in-depth investigation of the utility of large language models as text classifiers for biomedical article classification. The study uses several small and mid-size open source models, as well as selected closed source ones, and is more comprehensive than most prior work with respect to the scope of evaluated configurations: different types of prompts, output processing methods for generating both class and class probability predictions, as well as few-shot example counts and selection methods. The performance of the most successful configurations is compared to that of conventional classification algorithms. The obtained average PR AUC over 15 15 challenging datasets above 0.4 0.4 for zero-shot prompting and nearly 0.5 0.5 for few-shot prompting comes close to that of the naïve Bayes classifier ( 0.5 0.5 ), the random forest algorithm ( 0.5 0.5 with default settings or 0.55 0.55 with hyperparameter tuning) and fine-tuned transformer models ( 0.5 0.5 ). These results confirm the utility of large language models as text classifiers for non-trivial domains and provide practical recommendations of the most promising setups, including in particular using output token probabilities for class probability prediction. 1 Introduction In recent years, the development of deep learning applied to natural language processing has given rise to large language models (LLM) – neural networks based on the transformers architecture [ 55 ] , trained on huge text corpora to predict text continuation and often fine-tuned to chat or follow user instruction. Due to their emergent abilities, these models can be successfully used for tasks not only involving text generation, but also text understanding and reasoning [ 60 ] , without additional finetuning. These tasks include machine translation [ 65 ] , text summarization [ 66 ] , code generation [ 13 ] , question answering [ 53 ] , mathematical reasoning [ 28 ] , or text classification [ 52 , 11 , 59 , 21 ] . Using a large language model for any of such tasks requires designing an appropriate prompt, which may or may not include some examples of inputs and desired outputs. Prompts with one or more examples are referred to as one- or few-shot prompts whereas prompts with no examples are referred to as zero-shot prompts. 1.1 Motivation Text classification is one of the most commonly studied tasks at the intersection of natural language processing and machine learning [ 37 , 31 , 20 , 48 , 1 , 63 , 56 , 15 ] . The standard approach is to train a classification model from a training set of documents with class labels, transformed to a vector representation. It often yields satisfactory classification quality but is hardly applicable when labeled training documents are scarce or not available at all. This is the case, in particular, when class labels are obtained during a human annotation process, which is time consuming and costly. This obstacle is only partially alleviated by approaches reducing the amount of required labeled data, such as active learning [ 54 , 29 , 16 ] or self-training [ 47 ] . The primary appeal of LLMs as text classifiers is that they do not need labeled training data at all when used with zero-shot prompts, and with few-shot prompts they only need a very small number of labeled examples. What they require instead is that class labels are interpretable or class membership criteria can be specified in natural language. In several application domains these requirements are easier to satisfy than those of conventional machine learning algorithms. Even if labeled data for classification model creation is available or can be collected, LLMs can sometimes be an attractive alternative whenever “understanding” text contents and classes has more utility than generalizing observed relationship patterns between text representations and classes. This is likely to be the case for specialized domains that require some background knowledge as well as deeper text comprehension to properly assign classes. One such domain may be that of biomedical articles, where texts of all classes usually belong to the same thematic area and it may be hard to discern classes based just on word occurrence statistics. It is possible that large language models could utilize the knowledge encoded in billions of their parameters to achieve results as good as or better than conventional classifiers trained on labeled data. 1.2 Related Work With the onset of large language models, their increasing capabilities and accessibility stimulate a growing stream of publications reporting their use for text classification. Table 1 summarizes recent work that is the most closely related to the scope and aims of this article along the following dimensions: data: text datasets or classification tasks, models: LLMs used for text classification, prompts: types of prompts or prompt design techniques, example selection: methods used to select examples for few-shot prompts, evaluation: performance measures and evaluation procedures. Prompts are categorized as zero- or few-shot (one-shot in some cases) and their complexity is characterized. The use of prompt design techniques such as chain of thought (CoT) [ 61 ] or clue and reasoning prompting (CARP) [ 52 ] is also indicated where appropriate. When semantic similarity was used for example selection, the corresponding embedding model is provided. Whenever one- or few-shot prompts were used, the procedure used to separate training and test documents is specified in addition to classification quality measures in the evaluation column. An interesting aspect of study design is omitted in the table: the type of model output or output processing used. This is because the prior studies do not differ in output processing methods aside from minor variations, the most significant of which is using the JSON format or plain text for model answers. Nevertheless, this aspect is important to explain the novelty of this work and is therefore addressed in the discussion presented below. Table 1: Related work. Ref. Data Models Prompts Example Selection Evaluation [ 52 ] sentiment phrases, movie reviews, news articles GPT-3 DaVinci zero-shot, few-shot; vanilla, CoT, CARP random, SimCSE [ 22 ] and fine-tuned SimCSE embedding accuracy; train-test split [ 59 ] COVID-19 tweets, economic and e-commerce texts, SMS spam detection Llama-2-70B-Instruct, GPT-3.5 Turbo, GPT-4 zero-shot; vanilla - accuracy, F1 [ 11 ] tweets and Facebook comments on US elections FLAN-T5 XXL, Mistral-7B, Llama-3-8B, Llama-3-70B, GPT-3 Ada, GPT-3 DaVinci, GPT-4o (all except GPT-4o fine-tuned) zero-shot and one/few-shot (one-shot for threads); varying complexity random, with class balancing F1; bootstrapping [ 58 ] COVID-19 tweets, economic and e-commerce texts, SMS spam detection GPT-3.5 Turbo, GPT-4, Gemini-pro, Llama-3-8B Instruct (fine-tuned), Qwen-Chat 7B (fine-tuned), Qwen-Chat 14B, Vicuna v1.5 7B and 13B zero-shot, few-shot; vanilla manual accuracy, F1, uncertainty-error rate (custom metric of LLM’s refusals to answer); train-test split [ 21 ] tweets, news articles, movie reviews, Wikipedia comments, medical abstracts, legislation documents, safeguarding reports Llama-1-7B, Llama-2-7B, Flan-T5 780M, GPT-3.5 Turbo zero-shot, few-shot with one example per class; varying scope of task/domain description random, with class balancing F1; train-test split [ 26 ] healthcare-related tweets GPT-3.5, GPT-4 zero-shot; vanilla - precision, recall, F1; k-CV [ 33 ] medical article abstracts, COVID-19 literature, public health claims, discharge summaries GPT-3.5 Turbo, FLAN-T5 UL2, Tk-Instruct, Alpaca zero-shot, few-shot; vanilla PubMedBERT [ 25