Fine-Tuning Vision-Language Models for Neutrino Event Analysis in High-Energy Physics Experiments

Dikshant SagarDepartment of Computer ScienceUniversity of California, IrvineIrvine, CA 92697, Kaiwen YuDepartment of Computer ScienceUniversity of California, IrvineIrvine, CA 92697Alejandro YankelevichDepartment of PhysicsUniversity of California, IrvineIrvine, CA 92697Jianming BianDepartment of Ph · 2025

Read Fine-Tuning Vision-Language Models for Neutrino Event Analysis in High-Energy Physics Experiments by Dikshant SagarDepartment of Computer ScienceUniversity of California, IrvineIrvine, CA 92697, Kaiwen YuDepartment of Computer ScienceUniversity of California, IrvineIrvine, CA 92697Alejandro YankelevichDepartment of PhysicsUniversity of California, IrvineIrvine, CA 92697Jianming BianDepartment of Ph (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-19376.txt.

Introduction

Dikshant Sagar and co-authors fine-tune a vision-language model for neutrino physics in a 2025 paper. Particle detectors record interactions as pixelated images, and sorting those events is core work for experiments hunting the ghostly neutrino. The team adapted a large multimodal model, built on a recent open language model, to classify the images.

Its scores were compared with a standard convolutional network used by major experiments. The vision-language model matched or beat that baseline on accuracy, precision, recall, and a ranking measure. It also reasons in words, so extra text context can join the image evidence. The authors propose such models as a general backbone for event sorting across high-energy physics.

Catalog record

CREATORDikshant SagarDepartment of Computer ScienceUniversity of California, IrvineIrvine, CA 92697, Kaiwen YuDepartment of Computer ScienceUniversity of California, IrvineIrvine, CA 92697Alejandro YankelevichDepartment of PhysicsUniversity of California, IrvineIrvine, CA 92697Jianming BianDepartment of Ph
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE14 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-19376

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Fine-Tuning Vision-Language Models for Neutrino Event Analysis in High-Energy Physics Experiments Dikshant SagarDepartment of Computer ScienceUniversity of California, IrvineIrvine, CA 92697, Kaiwen YuDepartment of Computer ScienceUniversity of California, IrvineIrvine, CA 92697Alejandro YankelevichDepartment of PhysicsUniversity of California, IrvineIrvine, CA 92697Jianming BianDepartment of PhysicsUniversity of California, IrvineIrvine, CA 92697Pierre BaldiDepartment of Computer ScienceUniversity of California, IrvineIrvine, CA 92697 Abstract Recent progress in large language models (LLMs) [ 13 ] has shown strong potential for multimodal reasoning beyond natural language [ 34 ] . In this work, we explore the use of a fine-tuned Vision-Language Model (VLM), based on LLaMA 3.2 [ 19 ] , for classifying neutrino interactions from pixelated detector images in high-energy physics (HEP) experiments. We benchmark its performance against an established CNN baseline used in experiments like NOvA and DUNE [ 6 , 16 ] , evaluating metrics such as classification accuracy, precision, recall, and AUC-ROC. Our results show that the VLM not only matches or exceeds CNN performance but also enables richer reasoning and better integration of auxiliary textual or semantic context. These findings suggest that VLMs offer a promising general-purpose backbone for event classification in HEP, paving the way for multimodal approaches in experimental neutrino physics. 1 Introduction Machine learning has become integral to the physical sciences, especially in high-energy physics (HEP), where detectors produce vast, complex datasets [ 29 , 35 , 17 , 11 , 9 , 10 ] . Deep learning models are increasingly used to extract meaningful patterns, but their black-box nature often limits interpretability, an essential aspect in scientific analysis [ 8 ] . A common task, such as classifying neutrino events against background noise, has traditionally relied on reconstructed detector objects and hand-engineered features, which, while effective, can be limited by reconstruction errors and constrained representations [ 7 ] . This challenge parallels developments in computer vision, where the shift from handcrafted features to deep convolutional neural networks (CNNs) enabled direct learning from raw images and boosted performance [ 26 , 8 ] . Inspired by this success, HEP has begun adopting similar approaches, moving toward architectures that can process low-level detector data directly [ 5 , 11 , 9 , 10 ] . The emergence of Vision Language Models (VLMs), which are large networks pretrained on paired images and text, extends this trend further. VLMs can learn joint visual-semantic representations, offering improved classification capabilities along with interpretable, language-based explanations for predictions [ 36 ] . In this work, we explore fine-tuning VLMs for neutrino event classification. By leveraging their ability to extract features from raw detector inputs, we reduce dependence on engineered variables and demonstrate enhanced classification accuracy over conventional CNNs. Moreover, VLMs provide a richer interpretive layer by generating natural-language rationales, offering a promising path toward more transparent and informative machine learning models in experimental physics. In particular, we compare their performance against conventional CNNs and demonstrate that VLMs not only achieve superior classification accuracy but also provide a broader scope of reasoning and more informative explanations for their predictions. 2 Methods 2.1 Dataset We simulate a modular liquid argon time projection chamber (LArTPC) with 5\text{\,}\mathrm{m}\mathrm{m} pixel readout in a 2\text{\,}\mathrm{m} \times 2\text{\,}\mathrm{m} \times 7\text{\,}\mathrm{m} volume, featuring alternating anodes and cathodes along the x -axis with 0.3\text{\,}\mathrm{m} drift lengths. Using GENIE (v3.0.6) [ 4 , 3 ] , 190,000 electron neutrino ( \nu_{e} ) and muon neutrino ( \nu_{\mu} ) events with energies up to 10\text{\,}\mathrm{G}\mathrm{e}\mathrm{V} are generated with 74% interacting through the charged current (CC) and 26% through the neutral current (NC). Energy deposition is then simulated with GEANT4 (v11.2.0) [ 18 , 2 ] . XZ and YZ views are then downsampled to $5\text{\,}\mathrm{c}\mathrm{m}$\times$5\text{\,}\mathrm{c}\mathrm{m}$ pixels. Finally, we crop each event display to a 512\times 512 grayscale image (“pixel map”) centered on the interaction, creating the final dataset for training. 2.2 LLaMa 3.2 Vision Figure 1: LLaMa 3.2 Vision 11B finetuning pipeline. LLaMA Vision 3.2 is a family of VLMs developed by Meta [ 19 ] , extending the LLaMA 3.2 series with visual understanding capabilities. Unlike traditional CNNs designed solely for image-based tasks, LLaMA Vision integrates visual and textual modalities within a unified transformer architecture [ 32 ] . Trained on a broad corpus of images and documents, it can process inputs ranging from photographs and plots to pixelated detector data alongside natural language. The model employs a high-resolution vision encoder that tokenizes images into patch embeddings [ 15 ] , which are jointly processed with text tokens by a shared transformer decoder. This enables cross-modal contextual reasoning, making it particularly suited for tasks like neutrino event classification using sparse detector images. In this work, we fine-tune the 11B parameter version using supervised instruction tuning and QLoRA [ 14 ] , as outlined in Fig. 1 . This enables the model to adapt to physics-specific data while preserving its pretrained multimodal capabilities. A key advantage of this setup is the model’s ability to generate both predictions and natural-language justifications, improving interpretability and scientific insight. Through this study, we assess whether VLMs like LLaMA Vision 3.2 can offer competitive or complementary alternatives to traditional CNNs in high-energy physics applications. 2.2.1 Parameter Efficient Supervised Finetuning Fine-tuning VLMs like LLaMA Vision 3.2 [ 19 ] is computationally demanding due to their billions of parameters and memory footprint. Fully updating all model weights is often infeasible for domain-specific datasets, such as those in neutrino physics, which are typically small and risk overfitting or catastrophic forgetting [ 22 ] . To address this, we adopt a parameter-efficient fine-tuning (PEFT) strategy, using Quantized Low-Rank Adaptation (QLoRA) [ 14 ] . QLoRA combines two key techniques: (1) quantization, where base model weights are stored in 4-bit precision to reduce memory consumption; and (2) low-rank adaptation [ 22 ] , where small trainable matrices are added to attention and MLP layers, enabling effective task-specific learning without updating the full model. This approach allows us to fine-tune LLaMA Vision 3.2 11B [ 19 ] on our neutrino dataset using modest GPU resources, while preserving the model’s visual-linguistic capabilities. It enables faster experimentation, lower hardware demands, and makes large-model deployment more practical for high-energy physics research. 2.2.2 Model and Training Specifications We fine-tune the LLaMA 3.2 Vision Instruct 11B model. Its instruction-following capabilities allow us to frame neutrino event classification as a multimodal prompt-response task. Using the meta-llama/Llama-3.2-11B-Vision-Instruct checkpoint with 4-bit quantization via the BitsAndBytes library, we significantly reduce memory usage during training. Fine-tuning is conducted on four NVIDIA A6000 GPUs (49GB VRAM each) with a per-device batch size of 4. To enable efficient adaptation, we employ QLoRA [ 14 ] , which introduces low-rank trainable matrices into key network modules while keeping the base model frozen. Supervised fine-tuning is performed using Hugging Face’s SFTTrainer [ 33 ] on 190,000 labeled events over one epoch. The training setup includes an effective batch size of 8, the adamw_torch_fused optimizer with a learning rate of 2\times 10^{-4} , warm-up ratio of 0.03, and gradient clipping at 0.3. We use bfloat16 precision with TF32 fallback for numerical stability. Training was completed in approximately one week, with regular checkpointing and logging via TensorBoard. 2.2.3 Inference During inference, the model was prompted with a standardized system message outlining physics-specific context and interaction

Read the complete text