Coupled Particle Filters for Robust Affordance Estimation

Patrick Lowin, Vito Mengers, Oliver Brock · 2026

Read Coupled Particle Filters for Robust Affordance Estimation by Patrick Lowin, Vito Mengers, Oliver Brock (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-15223.txt.

Introduction

Lowin and co-authors estimate robot affordances with two coupled particle filters, one for graspable regions and one for movable regions. Single estimators struggle with visual, geometric, and semantic ambiguity. Each filter encodes regularities tied to its own property, which trims uncertainty. Coupling lets the two trade information and focus where both agree. On a real-world dataset, precision beat three recent estimators by more than two and a half times. Real robot trials reached a 70 percent success rate, even in low light and clutter.

Catalog record

CREATORPatrick Lowin, Vito Mengers, Oliver Brock
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31238
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.15223

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Coupled Particle Filters for Robust Affordance Estimation Patrick Lowin, Vito Mengers, Oliver Brock Abstract Robotic affordance estimation is challenging due to visual, geometric, and semantic ambiguities in sensory input. We propose a method that disambiguates these signals using two coupled recursive estimators for sub-aspects of affordances: graspable and movable regions. Each estimator encodes property-specific regularities to reduce uncertainty, while their coupling enables bidirectional information exchange that focuses attention on regions where both agree, i.e., affordances. Evaluated on a real-world dataset, our method outperforms three recent affordance estimators (Where2Act, Hands-as-Probes, and HRP) by 308%, 245%, and 257% in precision, and remains robust under challenging conditions such as low light or cluttered environments. Furthermore, our method achieves a 70% success rate in our real-world evaluation. These results demonstrate that coupling complementary estimators yields precise, robust, and embodiment-appropriate affordance predictions. I Introduction Affordances are the possible actions an agent can perform in its environment [ 7 ] . For robots that want to interact with the world, detecting affordances is essential yet challenging, as affordance cues are often ambiguous in the sensory input. Since robots can interact by grasping and moving objects, affordances arise in regions where these properties co-occur. In this paper, we present a robust method for detecting affordances by coupling estimators of graspable and movable regions and using their relationship to resolve ambiguities in the sensor data. Coupling the estimation processes for graspable and movable regions improves robustness in two ways. First, while we lack explicit affordance priors, we exploit available priors for graspability and movability to reduce uncertainty. For instance, predicted grasps in physically infeasible locations, such as edges caused by depth discontinuities, are rejected. Second, the relationship between the two properties provides an additional prior, allowing the estimators to focus on regions where both agree. This resolves ambiguities that a single estimator cannot. Initially, each estimator focuses on the regions most likely for its property. Through information sharing, they gradually align attention on regions important to both, i.e., the true affordances ( Figure 1 ). The strength of this coupling can be adjusted to control the interaction between the estimators. By coordinating in this way, they share complementary information, reduce uncertainty, and produce more robust affordance estimates. We evaluate our method on the RBO dataset [ 17 ] of articulated objects and interactions, which we extend with manipulable-region annotations. We compare against recent learning-based approaches, outperforming a geometry-based estimator trained in simulation by 308% and two appearance-based methods trained on human interactions by 245% and 257%. At the same time, our approach demonstrates improved robustness under challenging conditions such as low light or clutter. We further evaluate this robustness in real-world manipulation trials, achieving a 70% success rate. These results highlight that leveraging shared structure, complementary information, and coordinated attention through coupled estimation processes enables robust affordance estimation. Dataset annotations and code are available at https://github.com/tu-rbo/CoupledParticleFilters . Fig. 1: Robots can interact by grasping an object and then moving it. Consequently, affordances emerge in regions where these properties co-occur. By estimating these properties separately and leveraging their relationship, we can resolve ambiguities in both modalities and estimate robust affordances. II Related Work Affordance estimation is typically approached by learning affordances in simulation or from human interaction data within a single, monolithic model. We review these approaches in Section II-A , highlighting their struggles with out-of-distribution data and perceptual ambiguities. To address these challenges, we instead estimate affordances with two coupled recursive estimators for related properties, an approach that has already proven effective for other complex perceptual problems, as we review in Section II-B . II-A Affordance Estimation Affordances are characterized as an agent’s possible actions in an environment [ 7 ] . Their perception thus depends on the agent’s capabilities and tasks. Early works used hand-crafted, often geometry-based rules to predict object affordances [ 22 , 10 ] , but defining such rules for complex scenarios is infeasible. Recent learning-based methods capture general affordance features to enable interaction in simulation [ 21 , 24 , 30 , 29 ] , yet they often struggle to generalize to real-world environments due to the sim-to-real gap [ 28 ] . Other approaches learn from real-world human interaction videos [ 8 , 1 , 23 , 14 , 31 , 26 ] , but face visual ambiguities or embodiment mismatches. Moreover, they often assume different properties to be known in advance, such as actions [ 21 , 24 , 29 ] , objects [ 21 , 24 , 29 , 30 , 13 ] , movable parts [ 21 , 24 , 29 ] , or segmentation [ 21 , 24 , 29 , 30 , 13 ] . Our approach avoids these assumptions by factorizing affordances into two simpler properties: graspability and movability. While we also use a model trained on human data [ 8 ] to determine movability, we avoid embodiment mismatch by coupling with a robot-specific graspability model [ 27 ] . This coupling also aids in disambiguating affordance cues through complementarity of the two estimation processes. II-B Coupled Recursive Estimation Coupled recursive estimation encodes information about perceptual sub-aspects and their relationships, enabling robust interpretation of sensory signals through Bayesian inference. This approach has been applied to the estimation of kinematic structure and other dynamic properties of objects [ 16 ] as well as the overall object segmentation [ 18 ] . Its semi-decomposed structure allows integration of additional sensor modalities through simple composition, such as proprioception [ 15 ] , audio signals [ 3 ] , or hand poses of surrounding humans [ 25 ] . Unlike traditional distributed state estimation methods [ 5 , 11 ] , coupled recursive estimation enforces structured interactions between estimators, akin to factor graphs [ 6 ] but more specific. This structure constrains system dynamics, enabling complex action generation with simple gradient descent [ 19 ] , and shares similarities with biological perceptual processes, motivating its use as a model of human vision [ 20 , 2 ] . We adopt this framework for affordance estimation by coupling particle filters that operate on different signals and representations, graspability and movability. While prior work has explored coupling filters over the same state variables [ 12 ] , our approach coordinates multiple, distinct estimators via cross-modal fusion, as we explain in detail in the following. Algorithm 1 Coupled Particle Filters 1: Initialize: Two sets of particles \{x^{i}_{1}\}_{i=1}^{N} and \{x^{i}_{2}\}_{i=1}^{N} . 2: for each time step t do 3: for each Filter f\in\{1,2\} do 4: Prediction Step: Propagate particles according to optical flow. 5: Injection Step: Inject new particles in low-particle-density areas. 6: Update Step: Weight particles using neural network measurements. 7: end for 8: Cross-Modal Fusion: Weigh particles based on the other belief’s density. 9: for each Filter f\in\{1,2\} do 10: Resample: Draw particles from belief according to normalized weights. 11: end for 12: end for Fig. 2: Our approach estimates affordances as a combination of the graspable and movable regions of our environment. Since our measurement sources provide only uncertain measurements for these properties, we recursively estimate a belief for them and apply additional priors to resolve ambiguities. This aggregates a strong belief in regions with high measurements for their individual property.

Read the complete text