Text-promptable Object Counting via Quantity Awareness Enhancement

Miaojing Shi, Xiaowen Zhang, Zijie Yue, Yong Luo, Cairong Zhao, Li Li · 2025

Read Text-promptable Object Counting via Quantity Awareness Enhancement by Miaojing Shi, Xiaowen Zhang, Zijie Yue, Yong Luo, Cairong Zhao, Li Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06679.txt.

Introduction

Miaojing Shi and co-authors improve how vision-language models count objects named in a text prompt. Models often know the category but not the quantity. Their system, QUANet, adds quantity-focused prompts and a training loss that ties words to counts in the image.

A two-stream decoder blends a transformer path with a classic vision path, with adapters passing knowledge between them. A ranking loss keeps the two streams' counts in sane order. Tests on four standard sets show strong counting of unseen categories with no retraining. The code is public for others to build on.

Catalog record

CREATORMiaojing Shi, Xiaowen Zhang, Zijie Yue, Yong Luo, Cairong Zhao, Li Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-06679

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Text-promptable Object Counting via Quantity Awareness Enhancement Miaojing Shi, Xiaowen Zhang, Zijie Yue, Yong Luo, Cairong Zhao, Li Li Abstract Recent advances in large vision-language models (VLMs) have shown remarkable progress in solving the text-promptable object counting problem. Representative methods typically specify text prompts with object category information in images. This however is insufficient for training the model to accurately distinguish the number of objects in the counting task. To this end, we propose QUANet, which introduces novel quantity-oriented text prompts with a vision-text quantity alignment loss to enhance the model’s quantity awareness. Moreover, we propose a dual-stream adaptive counting decoder consisting of a Transformer stream, a CNN stream, and a number of Transformer-to-CNN enhancement adapters (T2C-adapters) for density map prediction. The T2C-adapters facilitate the effective knowledge communication and aggregation between the Transformer and CNN streams. A cross-stream quantity ranking loss is proposed in the end to optimize the ranking orders of predictions from the two streams. Extensive experiments on standard benchmarks such as FSC-147, CARPK, PUCPR+, and ShanghaiTech demonstrate our model’s strong generalizability for zero-shot class-agnostic counting. Code is available at https://github.com/viscom-tongji/QUANet 1 Introduction The object counting task was initially developed to estimate the quantity of specific targets within images or videos. Conventional methods achieve it by training deep neural networks to estimate object density maps on vast amounts of labeled data from predefined categories, such as crowds ( Liang et al. 2022 ) , vehicles ( Mundhenk et al. 2016 ) , and cells ( Xie et al. 2018 ) . Despite their achievements, these methods lack the flexibility to generalize to previously unseen categories. Recent methods introduce class-agnostic counting ( Lu et al. 2019 ; Ranjan and Nguyen 2022 ) , allowing for the counting of arbitrary categories by using exemplars as category indicators. By annotating a few image patches as exemplars and computing the similarities between exemplars and image regions, these methods have demonstrated promising accuracy as well as good generalizability. However, they still depend on the manual annotation for objects of interest, which is often inconvenient in real-world applications. Figure 1 : (a) Previous text-promptable object counting methods use simple text prompts and conventional vision-text contrastive loss. (b) Our QUANet introduces novel quantity-oriented text prompts with quantity alignment and ranking losses to enhance the model’s quantity awareness. Recently, pre-trained large vision-language models (VLMs) ( Liu et al. 2024 ; Li et al. 2023b ) have proven effective in numerous vision tasks. Inspired by their success, there has been a surge of interest in solving the class-agnostic counting problem based on VLMs, i.e . leveraging text prompts instead of exemplars to specify the target category of interest ( Kang et al. 2024 ; Jiang et al. 2023 ; Amini-Naieni et al. 2023 ; Zhai et al. 2025 ) . The text prompts are generated either manually ( Kang et al. 2024 ; Amini-Naieni et al. 2023 ) or automatically ( Doubinsky et al. 2024 ) . Both texts and images are encoded via the VLM to obtain their respective embeddings and then interacted for object count prediction. In this way, they achieve zero-shot object counting without visual exemplars. The performance of this text-promptable object counting highly relies on the quality of text prompts. Existing works use simple prompts or elaborate category-related details, for instance, using the color and position descriptions to specify target object categories ( Dai et al. 2024 ) . While this is important for object recognition, it is however insufficient for the model to distinguish the number of objects in the counting task. Paiss et al. 2023 instead alters the numerical information in text prompts to predefined numbers (up to 10), so as to leverage them as positive/negative vision-text pairs in the contrastive learning ( Paiss et al. 2023 ; Liang et al. 2023 ) . This reassures the model’s emphasis on the quantity side, yet is not enough. It only allows the model to coarsely know that these numerical values in texts correspond to different numbers of objects in images. Knowing how much is the difference between numbers, as the key to exact object counting, was however untouched in previous works. To achieve it, we propose novel quantity-oriented text prompts to dynamically manipulate the numerical information, and through a series of quantity alignment and ranking losses, we optimize the model to comprehensively understand object quantities in images (Fig. 1 ). Next, many object counting methods have chosen the Transformer-based encoders for robust feature encoding, yet when it comes to the object density estimation stage, CNN-based decoders remain the only choice ( Jiang et al. 2023 ; Amini-Naieni et al. 2023 ; Liu et al. 2022a ) . CNNs are effective in extracting and processing local features, which can be especially beneficial to density estimation-based object counting. The Transformer ( e.g . ViT ( Dosovitskiy et al. 2020 ) ), in spite of its global feature modeling capability, has not been successfully adopted in object density estimation ( Liu et al. 2022a ; Kang et al. 2024 ) . This can be attributed to the complication of the density estimation task, where the complementary characteristics between CNN and Transformer are imperfectly reflected in individual density maps. To cope with it, we empower a novel dual-stream adaptive counting decoder, where several Transformer-to-CNN enhancement adapters (T2C-adapters) and a gating network are specifically developed to transform the complementary nature between CNN and Transformer into realistic benefit for object counting. In this paper, we introduce the QUANet , a QU antity- A ware neural Net work for text-promptable object counting. During network training, given an image, we craft the quantity-oriented text prompts that consist of the factual text prompt specifying the exact object count and counterfactual text prompts specifying incorrect counts according to dynamically assigned intervals. Next, we pass these prompts and the image into the VLM’s encoders to obtain textual and visual embeddings, respectively. A dual-stream adaptive counting decoder is proposed to decode the density map, which consists of a Transformer (ViT) stream and a CNN stream to capture complementary features. We design several T2C-adapters and a gating net to enable effective knowledge communication and aggregation among these features, ultimately improving the counting performance. In the end, we propose the vision-text quantity alignment loss to enforce similarities between vision-text pairs to conform to specific quantity order given in text prompts; this is to inject the awareness of quantity differences into the visual counter. The idea is further reinforced via a new cross-stream quantity ranking loss, which enforces the patch-level counting consistency and ranking across the two streams of the decoder. During testing, quantity-oriented text prompts are unavailable; instead, a conventional category-specific text prompt is used. Extensive experiments on FSC-147 ( Ranjan et al. 2021 ) , CARPK ( Hsieh et al. 2017 ) , PUCPR+ ( Hsieh et al. 2017 ) , and ShanghaiTech ( Zhang et al. 2016 ) datasets demonstrate that our method outperforms state-of-the-art text-promptable object counting methods by a large margin. 2 Related work 2.1 Class-specific Object Counting Class-specific object counting focuses on counting specific categories from given images. These categories mainly include crowds ( Ranasinghe et al. 2024 ; Guo et al. 2024 ; Peng and Chan 2024 ; Wu and

Read the complete text