Rethinking Epistemic and Aleatoric Uncertainty for Active Open-Set Annotation: An Energy-Based Approach
Chen-Chen Zong, Sheng-Jun Huang · 2025
Read Rethinking Epistemic and Aleatoric Uncertainty for Active Open-Set Annotation: An Energy-Based Approach by Chen-Chen Zong, Sheng-Jun Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-19691.txt.
Introduction
Chen-Chen Zong and Sheng-Jun Huang tackle a labeling problem from 2025: how to pick the best items to label when some items belong to classes the model has never seen. Past methods pick either safe known items with little new value, or odd items whose scores mean little.
Their method uses two models and energy scores that blend two kinds of doubt. It first keeps a small pool of likely known items, then picks the ones the main model finds most unsure, and it adjusts the pool size as it goes. Tests show top scores with high pick accuracy and low added cost.
Catalog record
| CREATOR | Chen-Chen Zong, Sheng-Jun Huang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-19691
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rethinking Epistemic and Aleatoric Uncertainty for Active Open-Set Annotation: An Energy-Based Approach Chen-Chen Zong, Sheng-Jun Huang Abstract Active learning (AL), which iteratively queries the most informative examples from a large pool of unlabeled candidates for model training, faces significant challenges in the presence of open-set classes. Existing methods either prioritize query examples likely to belong to known classes, indicating low epistemic uncertainty (EU), or focus on querying those with highly uncertain predictions, reflecting high aleatoric uncertainty (AU). However, they both yield suboptimal performance, as low EU corresponds to limited useful information, and closed-set AU metrics for unknown class examples are less meaningful. In this paper, we propose an Energy-based Active Open-set Annotation (EAOA) framework, which effectively integrates EU and AU to achieve superior performance. EAOA features a ( C + 1 ) (C+1) -class detector and a target classifier, incorporating an energy-based EU measure and a margin-based energy loss designed for the detector, alongside an energy-based AU measure for the target classifier. Another crucial component is the target-driven adaptive sampling strategy. It first forms a smaller candidate set with low EU scores to ensure closed-set properties, making AU metrics meaningful. Subsequently, examples with high AU scores are queried to form the final query set, with the candidate set size adjusted adaptively. Extensive experiments show that EAOA achieves state-of-the-art performance while maintaining high query precision and low training overhead. The code is available at this link . 1 Introduction The success of deep neural networks (DNNs) is largely fueled by large-scale, accurately labeled datasets [ 16 , 49 ] . However, acquiring such data is often expensive and time-consuming primarily due to the labor-intensive nature of manual labeling [ 30 , 50 ] . To save labeling costs, recent research has focused on developing effective model training techniques that leverage limited and insufficiently labeled data [ 46 , 37 , 30 ] . Among them, active learning (AL) has emerged as a popular framework, which iteratively selects the most informative examples from the unlabeled data pool and queries their labels from the Oracle [ 34 , 30 , 12 ] . Figure 1 : Dataset: CIFAR-10; mismatch ratio: 40%. Our motivation: in open-set scenarios, querying examples with low epistemic uncertainty yields high query precision, but the overall information content is low, resulting in poor model performance. Focusing on examples with high aleatoric uncertainty also leads to poor performance, as the model’s assessment of this uncertainty becomes meaningless for open-set examples. Nevertheless, an effective combination of the two can lead to a superior outcome. Existing AL methods can be categorized into three types based on their sampling strategies: uncertainty-based [ 2 , 10 , 42 , 50 ] , diversity-based [ 25 , 33 , 40 ] , and hybrid strategies [ 11 , 1 , 31 ] . Most of these methods operate under the closed set assumption, which posits that the classes in the unlabeled data match those in the labeled data. However, in real-world scenarios, it is challenging and often costly to ensure that no open-set classes are present in the unlabeled data. Meanwhile, several studies [ 6 , 26 , 28 , 31 ] have shown that these methods perform poorly when open-set classes are involved, as such examples tend to receive uncertain model predictions and exhibit distinct features. Therefore, developing effective AL methods for open-world scenarios, where open-set classes exist, is of significant importance. Recently, this emerging research problem has garnered considerable attention [ 6 , 26 , 28 , 31 , 50 ] . For instance, in LfOSA [ 26 ] , the authors formulate this problem as open-set annotation (OSA) and utilize queried unknown class examples to train a (C+1) -class detector for rejecting open-set examples and focusing more on selecting known class ones. Two recent methods, EOAL [ 31 ] and BUAL [ 50 ] , adopt a structure similar to LfOSA, with EOAL aimed at improving the recognition of known class examples, while BUAL focuses more on sampling highly uncertain examples. Despite demonstrating high query precision, our findings reveal that these methods can not achieve satisfactory test accuracy and struggle to identify the most informative examples. To analyze the reasons behind their failure, we review the concept of uncertainty quantification 1 1 1 A detailed introduction with intuitive illustrations is in Appendix A . . Epistemic uncertainty refers to a measure that remains high for instances not previously encountered and decreases when these instances are included in the training [ 14 , 35 ] . In contrast, aleatoric uncertainty is characterized by elevated values in ambiguous examples [ 14 , 35 ] . In closed-set settings, examples with high epistemic uncertainty tend to reside in low-density regions of the representation space, while those with high aleatoric uncertainty may appear around the decision boundary due to ambiguous features. Both types are potential targets for our queries. However, in open-set scenarios, examples with high epistemic uncertainty are likely to be open-set instances, and aleatoric uncertainty is meaningful only in closed-set contexts, as it reflects the ambiguity between different observable classes 2 2 2 Essentially, the class probability p(y|x)=\frac{p(x,y)}{p(x)} is meaningful only when p(x)\neq 0 , i.e., x must be an observing example from a known class. [ 14 , 22 ] . Therefore, selecting examples with low epistemic uncertainty (to ensure a closed set) and high aleatoric uncertainty is a more reasonable choice. However, LfOSA and EOAL focus on querying examples with low epistemic uncertainty, while BUAL prioritizes querying those with high aleatoric uncertainty. This may be the potential reason for their failure. To validate this, we present the results in Figure 1 . As shown, focusing solely on either epistemic or aleatoric uncertainty in an open-world scenario results in suboptimal performance. However, effectively combining both leads to a significant improvement. Inspired by this, we propose E nergy-based A ctive O pen-set A nnotation (EAOA), an approach that effectively queries the most informative examples by considering both types of uncertainty. EAOA maintains two networks: a (C+1) -class detector and a C -class target classifier, with the following contributions: • An energy-based epistemic uncertainty measure is designed for the detector, expressed as the free energy score on known classes minus that on the unknown class. This measure integrates both learning-based and data-driven perspectives, enabling reliable uncertainty assessment in data-limited scenarios. • An energy-based aleatoric uncertainty measure is proposed for the target classifier, defined as the free energy score on all classes minus that on secondary classes. • A coarse-to-fine querying strategy is proposed. It first selects examples with low epistemic uncertainty to form a smaller candidate set, ensuring closed-set properties, which makes aleatoric uncertainty meaningful. Then, it queries examples with high aleatoric uncertainty within this set, with the candidate set size adaptively adjusted through a novel target-driven strategy. • A margin-based energy loss is introduced for the detector training, aimed at maximizing the free energy score on known classes while minimizing that on the unknown class, thereby enhancing the unknown class detection. • Extensive experiments show that EAOA outperforms current state-of-the-art methods in test accuracy, query precision, and training efficiency. 2 Related Work Active learning (AL) has garnered great research interest as a primary framework for