RISE: Enhancing VLM Image Annotation with Self-Supervised Reasoning

Suhang Hu, Wei Hu, Yuhang Su, Fan Zhang · 2025

Read RISE: Enhancing VLM Image Annotation with Self-Supervised Reasoning by Suhang Hu, Wei Hu, Yuhang Su, Fan Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-13229.txt.

Introduction

Suhang Hu and co-authors present RISE, a 2025 framework that teaches vision-language models to reason before labeling images. Hard annotation tasks, like emotion classes or context-based detection, demand real reasoning. Standard fine-tuning teaches only the final label, ignoring the rationale behind it.

Reinforcement approaches try to add reasoning but produce inconsistent chains of thought. RISE runs in two stages, first building verified reasoning traces, then strengthening the model with them. Its name spells out the path: reason, inspire, strengthen, expertise. Better reasoning should lift annotation quality on the tasks where labels alone fail.

Catalog record

CREATORSuhang Hu, Wei Hu, Yuhang Su, Fan Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-13229

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RISE: Enhancing VLM Image Annotation with Self-Supervised Reasoning Suhang Hu, Wei Hu, Yuhang Su, Fan Zhang Abstract Vision-Language Models (VLMs) struggle with complex image annotation tasks, such as emotion classification and context-driven object detection, which demand sophisticated reasoning. Standard Supervised Fine-Tuning (SFT) focuses solely on annotation outcomes, ignoring underlying rationales, while Visual Reinforcement Fine-Tuning (Visual-RFT) produces inconsistent Chains of Thought (CoTs) due to the absence of high-quality, verified CoTs during pre-training. We introduce RISE (Reason-Inspire-Strengthen-Expertise), a two-stage framework to overcome these limitations. In the Reason stage (RISE-CoT), a reinforcement learning-driven ” annotation-reasoning-annotation ” closed-loop generates visually grounded, logically consistent CoTs by verifying their ability to reconstruct original annotations without direct leakage. The Inspire and Strengthen stage (RISE-R1) leverages a high-quality CoT subset, filtered by RISE-CoT rewards, for supervised fine-tuning, followed by reinforcement fine-tuning to produce interpretable reasoning and accurate annotations, achieving Expertise in complex visual tasks. Evaluated on complex and simple image annotation tasks, RISE-trained Qwen2-VL-2B outperforms SFT and Visual-RFT, achieving robust performance and enhanced explainability. RISE offers a self-supervised solution for advancing VLM reasoning without requiring manually annotated CoTs. Code and resources are available at: https://github.com/HSH55/RISE. Introduction Image annotation , the process of assigning descriptive labels or semantic information to image elements, includes tasks like classification, detection, and segmentation. Traditional vision models, such as convolutional neural networks (CNNs) like ResNet ( He et al. 2015 ) and YOLO ( Redmon et al. 2016 ) , and early Transformer-based models like Vision Transformer (ViT) ( Dosovitskiy et al. 2021 ) , excel at capturing color and texture patterns but struggle with tasks requiring complex reasoning, such as emotion classification in Emotion6 ( Peng et al. 2015 ) (e.g., discerning “ joy ” from smiling faces and festive contexts) or context-driven object detection in LISA ( Lai et al. 2024 ) (e.g., identifying “ a wheel not in contact with the ground ”). Vision-Language Models (VLMs), such as GPT-4v ( OpenAI 2023 ) , Qwen-VL ( Bai et al. 2023 ) , LLaVA ( Liu et al. 2023 ) , and Gemini ( Google 2023 ) , integrate visual perception with linguistic reasoning, offering superior solutions for such tasks. However, general-purpose VLMs, especially lower-capacity models, often require task-specific fine-tuning on relevant datasets to achieve robust performance. To enhance VLMs for complex image annotation, Supervised Fine-Tuning ( SFT ) is commonly used to predict annotations. However, SFT struggles with complex tasks, as it predicts outcomes without capturing underlying reasoning. The advent of Chains of Thought (CoT) —textual narratives articulating step-by-step reasoning—has shown immense potential ( Wei et al. 2022 ; Jiang et al. 2025 ; Deepseek Team 2023 ) to improve VLM reasoning performance. CoTs enable models to produce ”think-answer” outputs, enhancing both accuracy and transparency. A CoT ”The scene shows smiling faces, bright colors, and festive settings ” leads to the annotation ” joy ”. Visual Reinforcement Fine-Tuning ( Visual-RFT ) ( Liu et al. 2025 ) and VLM-R1 ( Shen et al. 2025 ) , inspired by Open-R1 and DeepSeek-R1 ( Deepseek Team 2023 ) , uses reinforcement learning to produce CoTs, improving over SFT. Yet, as most datasets lack high-quality CoTs, Visual-RFT and VLM-R1 miss critical CoT pre-training, akin to DeepSeek-Zero ( Deepseek Team 2023 ) , and may generate superficial CoTs (e.g., “ This is a joyful scene ” \rightarrow “ joy: 1.0 ”). Our experiments and MME-CoT ( Jiang et al. 2025 ) confirm that low-quality CoTs reduce accuracy and generalization in complex tasks. High-quality CoTs are essential for initializing (warming up) VLMs to excel in reasoning tasks, as demonstrated by DeepSeek-R1 ( Deepseek Team 2023 ) . However, generating and validating such CoTs for image annotation datasets is challenging, as manual annotation is labor-intensive and large VLMs might produce inconsistent outputs. We hypothesize that A high-quality CoT should intrinsically encapsulate all necessary visual and contextual cues to derive the correct annotation , such as “The scene shows smiling faces, bright colors, and festive settings“ indicates “joy“ for Emotion6. To address these challenges, we propose RISE (Reason-Inspire-Strengthen-Expertise), a two-stage framework that self-supervises CoT generation and trains VLMs to produce interpretable “think-answer” outputs, enhancing reasoning and annotation accuracy for complex tasks. The Reason stage ( RISE-CoT ) is the cornerstone, employing a reinforcement learning-driven closed-loop ”annotation-reasoning-annotation” process. Here, a VLM generates a CoT from an image and its ground-truth annotation of a training sample, then reconstructs the annotation from the generated CoT. The CoT is quantified as a score by the accuracy of this reconstruction, providing a robust self-supervised signal for optimization via Group Relative Policy Optimization (GRPO) ( Azerbayev et al. 2024 ) . This process ensures visually grounded and logically consistent CoTs. Crucially, the CoT generation is designed to prevent direct leakage of annotations, forcing the VLM to genuinely describe the reasoning. The output of RISE-CoT is an enriched dataset, where each training sample is augmented with a CoT and its corresponding quality score, forming a comprehensive resource for subsequent fine-tuning. The Inspire and Strengthen stage ( RISE-R1 ) leverages this enriched dataset to optimize a VLM through two sequential steps. First, SFT is performed on a high-quality CoT subset, filtered based on RISE-CoT’s quality scores, to initialize the VLM. Second, Reinforcement Fine-Tuning (RFT) is applied on the entire dataset, further optimizing the VLM to produce precise annotations alongside interpretable CoTs. This VLM is deployed for image annotation tasks . Implemented with Qwen2-VL-2B, RISE demonstrably outperforms standard SFT and Visual-RFT in image annotation tasks. Our experiments highlight RISE’s ability to enhance VLMs’ reasoning by generating and leveraging high-quality CoTs in a self-supervised manner, ultimately underscoring the critical importance of verifiable, high-quality reasoning in multimodal learning. Related Works Traditional deep learning models , such as CNNs ( LeCun et al. 1989 ) , YOLO ( Redmon et al. 2016 ) for object detection, and UNet ( Ronneberger, Fischer, and Brox 2015 ) for segmentation, excel at pattern-based image analysis but lack natural language expression and complex reasoning capabilities. This limits the adaptability of traditional models to tasks requiring contextual understanding, such as emotion classification in Emotion6 ( Peng et al. 2015 ) or context-driven detection in LISA ( Lai et al. 2024 ) . Our RISE framework integrates visual and linguistic reasoning to produce interpretable outputs for such complex tasks. Vision-Language Models (VLMs) have transformed image understanding by combining visual perception with linguistic reasoning. CLIP ( Radford et al. 2021 ) achieves robust image-text alignment via contrastive learning, enabling zero-shot capabilities. Advanced models like GPT-4V ( OpenAI 2023 ) , Qwen-VL ( Bai et al. 2023 ; Qwen Team 2024 ) , LLaVA ( Liu et al. 2023 ; Liu, Li, and Lee 2024 ) , and Gemini ( Google 2023 ; Google 2024 ) excel in tasks like Visual Question Answering and image captioning. However, VLMs often struggle with complex image annotation tasks, such as object detection ( Lin et al. 2014 ) , requiring extensive fine-tuning with task-specific data. RISE enhances small VLMs by self-supervising high-quality reasoning generation, reducing reliance on annotated datasets. Reasoning-enhanced models introduce logical depth to AI systems. Chain of Thought (CoT) prompting ( Wei et al. 2022 ) enables step-by-step reasoning, advanced by DeepSeek-R1 ( Deepseek Team 2023 ) through reinforcement

Read the complete text