Radio Astronomy in the Era of Vision-Language Models: Prompt Sensitivity and Adaptation
Mariia Drozdova, Erica Lastufka, Vitaliy Kinakh, Taras Holotyak, Daniel Schaerer, Slava Voloshynovskiy · 2025
Read Radio Astronomy in the Era of Vision-Language Models: Prompt Sensitivity and Adaptation by Mariia Drozdova, Erica Lastufka, Vitaliy Kinakh, Taras Holotyak, Daniel Schaerer, Slava Voloshynovskiy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-02615.txt.
Introduction
Mariia Drozdova and five co-authors wrote a 2025 astronomy AI paper testing whether general vision-language models can classify radio galaxy shapes without astronomy training. The MiraBest dataset, with its two fan-shaped galaxy classes, supplies the test images.
Prompt styles varied across text descriptions, schematic diagrams, and visual examples placed inside the prompt, a first for astronomy work of this kind. Prompt-only attempts already did well, suggesting useful built-in priors. But outputs swung sharply with layout, ordering, or decoding settings, even when meaning stayed fixed. Light fine-tuning of one model, with few trainable parameters, nearly matched specialist state-of-the-art accuracy. The authors warn that apparent reasoning may reflect prompt sensitivity, while still seeing promise in adapted general models.
Catalog record
| CREATOR | Mariia Drozdova, Erica Lastufka, Vitaliy Kinakh, Taras Holotyak, Daniel Schaerer, Slava Voloshynovskiy |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-02615
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Radio Astronomy in the Era of Vision-Language Models: Prompt Sensitivity and Adaptation Mariia Drozdova, Erica Lastufka, Vitaliy Kinakh, Taras Holotyak, Daniel Schaerer, Slava Voloshynovskiy Abstract Vision–Language Models (VLMs), such as recent Qwen and Gemini models, are positioned as general-purpose AI systems capable of reasoning across domains. Yet their capabilities in scientific imaging, especially on unfamiliar and potentially previously unseen data distributions, remain poorly understood. In this work, we assess whether generic VLMs, presumed to lack exposure to astronomical corpora, can perform morphology-based classification of radio galaxies using the MiraBest FR-I/FR-II dataset. We explore prompting strategies using natural language and schematic diagrams, and, to the best of our knowledge, we are the first to introduce visual in-context examples within prompts in astronomy. Additionally, we evaluate lightweight supervised adaptation via LoRA fine-tuning. Our findings reveal three trends: (i) even prompt-based approaches can achieve good performance, suggesting that VLMs encode useful priors for unfamiliar scientific domains; (ii) however, outputs are highly unstable, i.e. varying sharply with superficial prompt changes such as layout, ordering, or decoding temperature, even when semantic content is held constant; and (iii) with just 15M trainable parameters and no astronomy-specific pretraining, fine-tuned Qwen-VL achieves near state-of-the-art performance (3% Error rate), rivaling domain-specific models. These results suggest that the apparent “reasoning” of VLMs often reflects prompt sensitivity rather than genuine inference, raising caution for their use in scientific domains. At the same time, with minimal adaptation, generic VLMs can rival specialized models, offering a promising but fragile tool for scientific discovery. 1 Introduction Vision–language models (VLMs), such as recent Gemini comanici2025gemini , Qwen-VL team2024qwen2 , and GPT-4o openai_chatgpt_2023 , show strong performance on general multimodal tasks, but their utility in scientific settings, especially when applied directly to domain-specific imagery, remains an open question. In astronomy, vision foundation models (VFMs) pretrained on natural images show promise when fine-tuned on scientific data lastufka2024vision ; drozdova2024semi . Even stronger results come from domain-specific pretraining, such as the vision-only model slijepcevic2024radio (which we refer to as AstroVFM), the multimodal AstroM3 rizhko2025astrom3 and CosmoCLIP imam2024cosmoclip . This raises the question: Can general-purpose VLMs adapt to scientific image classification with little or no supervision? VLMs have been explored in astronomy by zaman2025astrollava , which focuses on optical image captioning and visual question answering. We instead focus on binary morphology classification using the MiraBest dataset porter2023mirabest : radio galaxies classified into Fanaroff–Riley types (FR-I vs FR-II). Riggi et al. riggi2025evaluating evaluated VLMs on MiraBest, reporting low performance (e.g., \sim 30% F1 for Qwen2-VL-2B, \sim 20% for 7B) using base models in a chat interface. Their fine-tuned LLaVA model also remained below 30%. In contrast, we use the instruction-tuned Qwen2-VL-7B-Instruct and show that prompt design alone yields 84% Macro-F1, and LoRA tuning pushes it to 97%. We explore several prompting strategies : natural language descriptions, schematic diagrams, and, to the best of our knowledge for the first time in this domain, retrieval-augmented prompting. Inspired by the success of retrieval-augmented generation (RAG) in LLMs lewis2020retrieval and recent theory showing that in-context examples can act like implicit weight updates dherin2025learning , we retrieve support images in CLIP space radford2021learning and embed them in the prompt. In zero-shot settings (see also tanoglidis2024first ), Gemini achieves errors as low as 14%. Open models perform worse without examples, but improve notably when conditioned on retrieved ones. However, results reveal high sensitivity to prompt layout, decoding temperature, and example ordering, as presented in Sec. 3 . Finally, we fine-tune Qwen2-VL-7B-Instruct using LoRA hu2022lora . On the full training set, this reduces the error to 3%, which is close to the 2% error achieved by a domain-pretrained ResNet (AstroVFM) slijepcevic2024radio , self-superised through BYOL grill2020bootstrap on the RGZ DR1 dataset wong_2024_14195049 and also fine-tuned on MiraBest. Our results suggest that generalist VLMs encode useful representations for radio astronomy imagery, but their success depends critically on the prompt construction and adaptation method. (a) Diagram (b) FR-I (c) FR-II Figure 1 : FR-I vs FR-II radio galaxy morphologies. (a) Schematic illustration of the Fanaroff–Riley classification alexander_resources . (b–c) MiraBest radio images. 2 Methods Task and Dataset. We study morphology-based binary classification of radio galaxies into Fanaroff–Riley types I and II fanaroff1974morphology . FR-I galaxies have bright central cores with fading jets (Fig. 1(b) ); FR-II exhibit edge-brightened lobes with prominent hotspots (Fig. 1(c) ). We use the MiraBest-confident split porter2023mirabest , a class-balanced set of 729 training and 104 test images from NVSS condon1998nrao and FIRST becker1994vla , labeled by experts. Although public and potentially seen during VLM pretraining, we treat this task effectively out-of-domain relative to typical natural image distributions. Models. We evaluate open-source VLMs, such as Qwen2/2.5-VL-Instruct team2024qwen2 , 2B–7B; LLaVA-1.6-Mistral liu2024llavanext , using chat-style prompting with structured system/user/assistant roles. Proprietary models, such as Gemini-2.5-Flash comanici2025gemini and GPT-4o openai_chatgpt_2023 , are accessed via API with fixed decoding settings; both use conversational interfaces. Prompting Setup. Each test query ends with a radio image and a prompt requesting classification as FR-I or FR-II. The context provided before the image varies across five prompting strategies: (1) Text , natural language descriptions of each class (zero-shot); (2) Diagram , the same text augmented with an abstract schematic (Fig. 1(a) ); (3) Fixed-Imgs , four labeled support images, identical across test queries; (4) kNN-Imgs , five labeled nearest neighbors retrieved per test sample in CLIP space radford2021learning ; and (5) kNN-Balanced , a balanced version of (4), retrieving labeled neighbors equally from both classes, a setup not commonly explored in prior VLM work. In strategies (3)-(5), both support images and their labels are included in the prompt. We evaluate each setup in two variants: chain-of-thought (CoT), prompting models to explain their reasoning, and noCoT, requesting only a direct label. To probe decoding variability, we vary temperature: lower values yield deterministic outputs, while higher ones increase sampling diversity, potentially revealing reasoning instability. Fine-Tuning. We explore whether lightweight adaptation improves VLM performance. We fine-tune Qwen2-VL-7B-Instruct using LoRA hu2022lora , updating \sim 15M parameters across vision and language modules. LoRA hyperparameters ( r=16 , \alpha=64 , dropout=0.3) are chosen via grid search. Training runs for 100 epochs, with test metrics reported at the first minimum of training loss. We compare results across varying training set sizes to a supervised ResNet trained from scratch and AstroVFM fine-tuned, both using MiraBest-confident as in slijepcevic2024radio . Our model is fine-tuned on the same dataset. 3 Results Figure 2 : Test error rates across prompting strategies with/without CoT. Boxplots summarize variation across prompts and image placement with respect to the query question. More details in Appendix B . (a) (b) (c) Figure 3 : Stability of example-conditioned prompts: (a) Error vs. number of retrieved neighbors; lower temperatures reduce error and spread; (b) Error across all permutations: Fixed-Imgs is less stable than kNN-Imgs ; (c) kNN-Imgs outperform kNN majority voting by 5 points across train sizes. Prompting Fig. 2 summarizes test error across prompting types. In Text and Diagram settings, Gemini performs best, reaching \sim 14% error. Open models lag behind: Qwen2-VL-7B-Instruct reaches \sim 28%, while LLaVA fails to generalize (>45%). A diagram rarely helps: only Gemini shows clear gains, suggesting its stronger capacity for abstract visual reasoning. Example-based prompts substantially improve Qwen’s performance. A simple CLIP-based kNN baseline