ProbRes: Probabilistic Jump Diffusion for Open-World Egocentric Activity Recognition
Sanjoy Kundu *, Shanmukha Vellamcheti, Sathyanarayanan N. Aakur · 2025
Read ProbRes: Probabilistic Jump Diffusion for Open-World Egocentric Activity Recognition by Sanjoy Kundu *, Shanmukha Vellamcheti, Sathyanarayanan N. Aakur (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-03948.txt.
Introduction
Sanjoy Kundu, Shanmukha Vellamcheti, and Sathyanarayanan Aakur tackle activity recognition from a head-worn camera when the set of possible activities is not fixed in advance. Real life is open-ended, and a system must find the right label in a huge space it has only partly seen. Writing in twenty twenty-five, they propose a probabilistic search that jumps and drifts through a space of candidate labels. Common-sense structure keeps the space sensible, vision-language models refine guesses, and randomness in the search avoids checking every label.
They test the system at four levels of openness, from a fixed label set to wide-open search, on kitchen, gaze, and home activity data sets. Performance led the field in their benchmarks at each level. A clear taxonomy of openness levels is offered alongside, to help later work state exactly which problem it solves.
Catalog record
| CREATOR | Sanjoy Kundu *, Shanmukha Vellamcheti, Sathyanarayanan N. Aakur |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-03948
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ProbRes: Probabilistic Jump Diffusion for Open-World Egocentric Activity Recognition Sanjoy Kundu *, Shanmukha Vellamcheti, Sathyanarayanan N. Aakur Abstract Open-world egocentric activity recognition poses a fundamental challenge due to its unconstrained nature, requiring models to infer unseen activities from an expansive, partially observed search space. We introduce ProbRes, a Probabilistic Residual search framework based on jump-diffusion that efficiently navigates this space by balancing prior-guided exploration with likelihood-driven exploitation. Our approach integrates structured commonsense priors to construct a semantically coherent search space, adaptively refines predictions using Vision-Language Models (VLMs) and employs a stochastic search mechanism to locate high-likelihood activity labels while minimizing exhaustive enumeration efficiently. We systematically evaluate ProbRes across multiple openness levels (L0–L3), demonstrating its adaptability to increasing search space complexity. In addition to achieving state-of-the-art performance on benchmark datasets (GTEA Gaze, GTEA Gaze+, EPIC-Kitchens, and Charades-Ego), we establish a clear taxonomy for open-world recognition, delineating the challenges and methodological advancements necessary for egocentric activity understanding. Our results highlight the importance of structured search strategies, paving the way for scalable and efficient open-world activity recognition. Code (in supplementary) will be shared publicly after review. 1 Introduction Figure 1 : Taxonomy of Openness in Egocentric Activity Recognition. We define four levels of openness based on search space constraints: L0 (fixed activity set), L1 (known atomic concepts, unknown compositions), L2 (known domain, inferred activities), and L3 (fully unconstrained search). Egocentric activity recognition in open-world settings presents a fundamental challenge in perception-driven reasoning, where an intelligent system must infer ongoing activities from a vast, unconstrained space of possibilities. Unlike closed-set classification, where predefined categories limit ambiguity, open-world recognition requires models to adaptively infer plausible activities, even when faced with unknown or unseen combinations of actions and objects. This challenge is particularly relevant for assistive robotics, wearable AI, and autonomous agents, where real-time understanding of human actions is crucial for interaction and decision-making. While multimodal foundation models such as Vision-Language Models (VLMs) have demonstrated remarkable zero-shot generalization, their reliance on exhaustive enumeration for inference makes them inefficient for large-scale open-world reasoning. To bridge this gap, structured knowledge is essential to move beyond naive brute-force enumeration, enabling scalable and efficient activity understanding in real-world applications. (a) (b) Figure 2 : (a) ProbRes framework for open-world egocentric activity recognition. The search space is structured using ConceptNet priors, enabling guided exploration. The model iteratively refines candidates via likelihood estimation, balancing exploration and exploitation, followed by local refinement and re-ranking. ( b) Search trajectory visualization , showing how ProbRes navigates likelihood regions to reach high-confidence predictions near the ground truth. The increasing focus on open-world learning [ 26 , 2 , 13 , 20 ] has led to ambiguity in defining openness, as different factors, objects, actions, and domains can independently contribute to it. Without a clear distinction, comparing methods and assessing generalization remain challenging. To address this, we define a hierarchy of openness levels that systematically capture increasing uncertainty in the search space (Figure 1 ). At L0 , as in traditional zero-shot learning, all possible activity categories (action-object combinations) are predefined, but some may lack training examples. This setting benefits directly from Vision-Language Models (VLMs), which can generalize to unseen instances but operate within a bounded conceptual space. At L1 , the atomic concept space — objects and actions — is known, but their composition (activities) is undefined, requiring models to reason about plausible pairings. At L2 , the domain is known (e.g., cooking), but the underlying search space is unconstrained. Finally, at L3 , no prior knowledge about the domain, actions, or objects is available, requiring models to construct the entire search space on the fly. As openness increases, exhaustive enumeration becomes intractable and requires structured priors for adaptive search. To address these, we introduce Probabilistic Residual Search (ProbRes), a structured search framework designed to navigate unconstrained activity spaces efficiently. Instead of relying solely on VLM-based likelihood estimation, ProbRes integrates structured priors from external knowledge sources to guide exploration, ensuring semantically meaningful search trajectories without supervision. As illustrated in Figure 2 , ProbRes operates in three key phases: (1) exploration, where prior-driven sampling identifies plausible activity candidates; (2) exploitation, where the search refines predictions based on VLM likelihoods; and (3) residual refinement, which further optimizes results by re-ranking activities based on their decomposed action-object components. By balancing structured priors with data-driven reasoning, ProbRes efficiently locates high-likelihood activities while drastically reducing VLM queries. Our contributions are four-fold: (i) we propose ProbRes , a probabilistic residual search strategy that integrates structured priors with likelihood-driven inference, effectively balancing exploration and exploitation for open-world activity recognition, (ii) we introduce a clear hierarchical taxonomy (L0–L3) to systematically define and evaluate open-world recognition challenges, providing a principled foundation for future research, (iii) we show that ProbRes drastically reduces VLM queries while maintaining or improving recognition accuracy, demonstrating the importance of structured priors in large-scale search, and (iv) we highlight the limitations of current VLM-based text embeddings for search efficiency with qualitative analysis and highlight the need for improved semantic structuring. 2 Related Work Egocentric video understanding has gained significant traction in the computer vision community in recent years due to its critical applications in areas such as imitation learning for robotic skills [ 45 ] , virtual reality [ 21 ] , and autonomous skill acquisition [ 22 , 27 , 4 ] . A wide range of tasks have been proposed to analyze egocentric video understanding, including procedure understanding, EgoPose [ 18 , 19 ] , proficiency estimation, question answering [ 15 ] , gaze prediction [ 1 , 29 , 16 ] , activity recognition [ 28 ] , and summarization [ 33 ] , driven by the development of large-scale datasets such as EgoExo4D [ 19 ] , Ego4D [ 18 ] , EPIC-Kitchens [ 10 ] , Charades-Ego [ 46 ] , GTEA Gaze [ 29 ] , GTEA Gaze Plus [ 16 ] , and Assembly101 [ 44 ] . Despite these advancements, action recognition in egocentric videos remains challenging, particularly under conditions of rapid camera movement and significant occlusion, leading to issues with accuracy and generalization across different environments. While early approaches to egocentric action recognition were dominated by supervised learning methods, such as spatial-temporal modeling [ 49 ] , motion-based representations [ 34 ] , hand-object interactions [ 53 , 62 , 56 ] , and time-series modeling [ 43 ] , recent research has shifted towards self-supervised learning [ 59 , 57 ] , leveraging large-scale pretraining followed by fine-tuning for downstream tasks. Both paradigms remain data-driven and require extensive training to achieve optimal performance. Multi-modal LLM-based models such as MM-Ego [ 58 ] and GPT4Ego [ 9 ] have recently been explored for egocentric action recognition. Zero-shot learning [ 46 , 60 ] offers a potential avenue to reduce dependence on labeled datasets, but its effectiveness is constrained by the inherent limitations of pre-trained models in reasoning about open-world activity compositions. Open-world understanding , particularly in egocentric activity recognition, remains an underexplored area. While significant progress has been made in open-world object detection, such as Open World DETR [ 13 ] and open-vocabulary object detection [ 20 , 14 ] , which leverage VLMs through prompting and knowledge distillation, challenges remain in handling