CountSteer: Steering Attention for Object Counting in Diffusion Models
Hyemin Boo, Hyoryung Kim, Myungjin Lee, Seunghyeon Lee, Jang-Hwan Choi, Hyunsoo Cho · 2025
Read CountSteer: Steering Attention for Object Counting in Diffusion Models by Hyemin Boo, Hyoryung Kim, Myungjin Lee, Seunghyeon Lee, Jang-Hwan Choi, Hyunsoo Cho (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-11253.txt.
Introduction
Hyemin Boo, Hyoryung Kim, and their colleagues present CountSteer, a training-free method that helps image generators draw the exact number of objects a prompt asks for. Diffusion models make realistic images but often miscount, which shows a gap between words and pictures. The authors found the models are not blind to counts: internal signals shift in steady ways when the output count is right or wrong.
CountSteer steers the model's cross-attention states during generation to use that latent count sense. In experiments, object-count accuracy rose by about four percent with no loss of visual quality. The authors present it as a simple step toward more controllable, reliable generation.
Catalog record
| CREATOR | Hyemin Boo, Hyoryung Kim, Myungjin Lee, Seunghyeon Lee, Jang-Hwan Choi, Hyunsoo Cho |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-11253
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CountSteer: Steering Attention for Object Counting in Diffusion Models Hyemin Boo, Hyoryung Kim, Myungjin Lee, Seunghyeon Lee, Jang-Hwan Choi, Hyunsoo Cho Abstract Text-to-image diffusion models generate realistic and coherent images but often fail to follow numerical instructions in text, revealing a gap between language and visual representation. Interestingly, we found that these models are not entirely blind to numbers—they are implicitly aware of their own counting accuracy, as their internal signals shift in consistent ways depending on whether the output meets the specified count. This observation suggests that the model already encodes a latent notion of numerical correctness, which can be harnessed to guide generation more precisely. Building on this intuition, we introduce CountSteer , a training-free method that improves generation of specified object counts by steering the model’s cross-attention hidden states during inference. In our experiments, CountSteer improved object-count accuracy by about 4% without compromising visual quality, demonstrating a simple yet effective step toward more controllable and semantically reliable text-to-image generation. Figure 1: CountSteer improves numerical fidelity in text-to-image diffusion. While baseline Stable Diffusion v1.5 often fails to generate the correct number of objects, our cross-attention steering framework generates images with more accurate object counts that better align with user-specified prompts across diverse object categories. Figure 2: CountSteer: During each denoising step t , adaptive steering vectors are injected into UNet blocks to steer hidden states based on latent distributional differences between correct ( \mu^{1}_{t,b} ) and incorrect ( \mu^{0}_{t,b} ) samples. Introduction Recent progress in text-to-image (T2I) diffusion models such as Stable Diffusion ( 10 ) has expanded the frontier of generative AI, enabling models to create realistic and semantically rich images from natural language descriptions. These models have been applied across diverse domains, including creative design, simulation, education, and synthetic data generation for downstream AI training, serving as powerful tools that bridge text and visual expression through controllable and cost-efficient synthesis. The effectiveness of diffusion models fundamentally depends on fidelity, which encompasses how faithfully generated images capture the intended semantics, structure, and quantitative details of the input text. While fidelity can be viewed through semantic, structural, and quantitative aspects, current models still face persistent challenges in maintaining fine-grained accuracy, particularly in object composition, color consistency, and object count ( 3 ; 12 ) . Among these factors, object count provides a clear and quantifiable measure of fidelity. However, diffusion models often struggle to maintain numerical accuracy, as they must correctly interpret and preserve count information throughout a stochastic and spatially complex generation process. This frequently results in under- or over-generation of objects, thereby reducing reliability in applications that demand semantic precision and quantitative accuracy. To gain deeper insight into this limitation, we hypothesize that diffusion models possess an internal understanding of numerical concepts but fail to effectively utilize this knowledge during image generation. We therefore examine how the model’s internal representations behave when it succeeds or fails to follow numerical instructions. Our analysis reveals that the latent distributions of cross-attention representations are distinguishable, exhibiting directional patterns that implicitly encode numerical awareness. These structured signals, which can be represented as a global vector in the attention space, indicate that diffusion models already contain latent cues related to counting behavior (see Section Numerical Steering via KDE Analysis for details). Building on this intuition, we introduce CountSteer , a training-free inference-time method that leverages these latent cues to enhance object-count fidelity. Unlike prior approaches that rely on fine-tuning ( 2 ; 11 ) or architectural modifications ( 4 ; 7 ) , our method operates directly during inference without retraining or altering the model structure. Specifically, CountSteer identifies a directional pattern within the model’s attention space that separates correct from incorrect generations and uses it to gently steer the diffusion trajectory toward numerically consistent outcomes. By modulating this steering signal adaptively during inference, CountSteer enhances count accuracy without altering model parameters or compromising image quality. Empirically, we observe an average 4% improvement in object-count fidelity, demonstrating that small, geometry-based interventions can unlock latent reasoning capabilities in large generative models. Figure 3: Visualization of hidden state distributions analyzed via KDE. The two classes (correct and incorrect counts) exhibit clearly separable regions, supporting the formulation of our steering direction vector. Preliminary Steering Steering is a training-free method to control a model’s latent feature space. During generation, the model tends to form distinct hidden-state distributions for correct (Class 1) and incorrect (Class 0) samples. We vectorize this distributional difference and use it as a steering direction to shift the model’s representation toward the correct distribution. This phenomenon was also observed in Inference-Time Intervention ( 6 ) , where hidden representations of language models exhibited linearly separable directions corresponding to truthful and untruthful classes. In ITI, interventions were applied to head-level activations of multi-head attention at each transformer layer. We adopt this concept to the structure of Stable Diffusion, performing interventions at each denoising step t within the UNet blocks b . Furthermore, we empirically examine whether a similar separation emerges in a diffusion model between Class 1 and Class 0. For each class, we compute the mean of hidden state vectors for the Class 1 and Class 0 samples, denoted as \mu^{1} and \mu^{0} . The difference \vec{s}=\mu^{1}-\mu^{0} is then defined as the steering direction vector, which guides the model toward improved output alignment: \mu_{1}=\frac{1}{N_{1}}\sum_{i=1}^{N_{1}}h_{i}^{(1)},~~\mu_{0}=\frac{1}{N_{0}}\sum_{j=1}^{N_{0}}h_{j}^{(0)} (1) h_{t}^{\prime}=h_{t}+\alpha\cdot\vec{s} (2) During inference, the hidden state h_{t} is adjusted by adding the steering direction \vec{s} . Here, \alpha controls the strength of the adjustment. We further extend this formulation across diffusion steps, defining a step and block-wise steering vector as \vec{s}_{t,b}=\mu^{1}_{t,b}-\mu^{0}_{t,b} and applying it dynamically throughout the denoising process. Numerical Steering via KDE Analysis We analyze the distributions in the image generation process with respect to object count fidelity. The samples are divided into Class 1 and Class 0. Visualization via Kernel Density Estimation (KDE) shows that the query hidden states of these two classes represent separable distributions. This observation suggests that, while the model inherently has the ability to represent object counts, such a capability is not consistently preserved during inference. Based on this observation, we hypothesize that steering the hidden representation of a generated sample to the distribution of Class 1 can improve count alignment. To achieve this, we define \vec{s}_{t,b} as the difference between the mean hidden states of Class 1 and Class 0, and apply it to adjust the latent representation during inference. This approach provides a training-free mechanism to improve numerical fidelity. Method Our analysis reveals that text-to-image diffusion models’ hidden states form two distinct distributions based on object count correctness. While steering vectors can shift representations between these distributions, prompt-dependent variability causes fixed vectors to yield insufficient or excessive corrections. We therefore introduce two adaptive scaling factors that modulate the steering vector’s magnitude and direction, enabling prompt-specific calibration. Figure 4: We generate images from ”{count} {object}” prompts, extract hidden states h_{t,b} , and manually annotate them into balanced classes of 200 correct and incorrect counting to construct the base steering vector. Dataset Construction As shown in Figure 5 of the Appendix, the model’s accuracy in producing the correct number of objects decreases as the target count increases. Beyond four, the generations almost entirely fail, suggesting that the model lacks consistent