The SET Perceptual Factors Framework: Towards Assured Perception for Autonomous Systems
Troi Williams · 2025
Read The SET Perceptual Factors Framework: Towards Assured Perception for Autonomous Systems by Troi Williams (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-10798.txt.
Introduction
Williams proposes the SET framework to assure perception in autonomous systems, in a paper from twenty twenty-five. Robots must perceive reliably to decide safely, yet failures often stem from common environmental factors. Weather, occlusion, and sensor limits can each break detection or pose estimation. The framework sorts such factors by source as Self, Environment, or Target.
State trees map where factors originate in each source class. Factor trees model how sources and factors degrade a given task. Combined models quantify uncertainty for that task in numbers. The aim is a transparent, standard way to name and share perception risks. Better assurance should build public trust in autonomous systems.
Catalog record
| CREATOR | Troi Williams |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 10 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-10798
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The SET Perceptual Factors Framework: Towards Assured Perception for Autonomous Systems Troi Williams Abstract Future autonomous systems promise significant societal benefits, yet their deployment raises concerns about safety and trustworthiness. A key concern is assuring the reliability of robot perception, as perception seeds safe decision-making. Failures in perception are often due to complex yet common environmental factors and can lead to accidents that erode public trust. To address this concern, we introduce the SET (Self, Environment, and Target) Perceptual Factors Framework. We designed the framework to systematically analyze how factors such as weather, occlusion, or sensor limitations negatively impact perception. To achieve this, the framework employs SET State Trees to categorize where such factors originate and SET Factor Trees to model how these sources and factors impact perceptual tasks like object detection or pose estimation. Next, we develop Perceptual Factor Models using both trees to quantify the uncertainty for a given task. Our framework aims to promote rigorous safety assurances and cultivate greater public understanding and trust in autonomous systems by offering a transparent and standardized method for identifying, modeling, and communicating perceptual risks. I Introduction Perceptual tasks like object detection typically seed planning and action [ 1 ] . Thus, autonomous systems, from autonomous cars to caregiving robots, must perceive their surroundings accurately to operate safely and reliably. However, perception often fails due to factors such as poor visibility, occlusion, or glare (Figure 1 ). These failures can distort a system’s belief of the world and lead to unsafe actions, (severe) accidents, and reduced public trust [ 2 , 3 , 4 ] . To mitigate perceptual failures and their consequences, it is crucial to quantify perception uncertainty—how uncertain a system is about its current perceptual predictions. The robotics community has studied perceptual uncertainty extensively (see Section III for a brief review) for decades. However, we lack a unifying framework to systematically identify, categorize, analyze, and communicate the factors causing perceptual failures, the sources of these factors, and their impact. Such a gap slows the development of robust safety assurances and makes it difficult for robotists, regulators, and the public to discuss perceptual risks. We propose the SET Perceptual Factors Framework to address this gap, where SET stands for Self (the perceiving agent), Environment, and Target. This framework contributes a structured, human-understandable approach to: 1) identify and model the perceptual factors (like glare) that impact perceptual tasks such as detection or localization; 2) determine how various sources (like the perceiving camera, the sun, and the target car) contribute to these factors; and 3) create Perceptual Factor Models (PFMs) that quantify perceptual uncertainty for given input factors or states. Fig. 1: Perceptual failures undermine safety and trust. For example, adverse weather due to haze (left) and snow (right) potentially caused YOLOv8 [ 5 ] to fail to detect vehicles (images from the DAWN dataset [ 6 , 7 ] ). If an autonomous car solely relied on these failed detections, the car could misjudge safety, potentially causing an accident (e.g., during a left turn), injuring occupants, and eroding public confidence. II The SET Perceptual Factors Framework The SET framework’s goal is to model how human-understandable factors and their sources impact perception uncertainty. The framework is composed of three components: a SET State Tree, a SET Factor Tree, and a PFM. II-A SET State Tree: The Detailed Sources of Uncertainty Fig. 2: This figure illustrates a sample SET State Tree. The tree models the origins of sources (objects or phenomena) that may impact the car detection task depicted in Figure 1 left. This tree also has the same sources depicted in the SET Factor Tree (Figure 3 ). Here, “self” denotes the perceiving agent’s camera, and the targets are the other cars on the road. We use bold boxes for sources and dashed boxes for variables describing the states of sources. A SET State Tree enumerates the sources (objects or phenomena) that produce perceptual factors and impact perception uncertainty (Figure 2 ). This tree helps us and a perceiving agent answer: What attributes of the world and myself (that is, sources and their states) could cause perceptual failures? We organize sources under three branches: • Self: The perceiving agent itself, including its sensors (for example, see “Self” in Figure 2 ) and other parts of the agent that might affect uncertainty (for example, reflective vehicle surfaces that may cause glare). • Environment: Any object or phenomena, excluding the perceiving agent (self) and the target, such as weather conditions (like rain or fog), light sources (like streetlights), and objects (like trees and buildings). • Target: The objects or phenomena the agent aims to perceive (for example, vehicles in Figure 1 ). II-B SET Factor Tree: The Chain of Uncertainty A SET Factor Tree models how sources and factors affect perception uncertainty (Figure 3 ), showing a chain—from sources to factors and from factors to the perceptual task—to answer two questions. First, how do sources create factors? And second, how do factors impact uncertainty quantitatively (that is, degrade a task)? We can use deterministic or probabilistic methods to model these relationships hierarchically: 1. Sources: The leaves of the tree, denoting the objects or phenomena derived from the State Tree. 2. Perceptual Factors: Intermediate nodes, denoting factors such as glare intensity and motion blur magnitude. These factors arise from interactions between sources. For example, glare is due to sunlight (Environment) shining into a camera (Self), causing image saturation that hinders detection of other vehicles (Target). 3. Perceptual Task: The root node of the tree, denoting the task we are modeling (e.g., object detection) and whose certainty is affected by perceptual factors. Fig. 3: This graph shows a sample SET Factor Tree for the object detection task in Figure 1 left. YOLOv8 may have failed to detect the car (target) due to perceptual factors such as motion blur, glare from the headlights, and (low) illumination. These factors could arise due to sources such as the camera, car, mist, and sun. We use dotted boxes to denote factor nodes and circles for variable nodes (observation, perceptual factors, and sources). II-C SET Perceptual Factor Models: Quantifying Uncertainty Fig. 4: This figure illustrates a sample SET Perceptual Factor Model for a car detection task based on Figure 1 . The model predicts if the perceptual algorithm, such as YOLOv8, will detect the car given the source states \lambda [ 8 , 9 , 10 ] . The input vector shows examples of source states. We now create a SET Perceptual Factor Model (PFM), which quantifies perception uncertainty based on the states of the sources (like [ 8 , 11 , 9 ] ) or the magnitude of the factors (like [ 10 ] ) (Figure 4 ). A PFM could be a neural network, Gaussian Process [ 12 ] , or another probabilistic model. Whether the input of a PFM is states or factors, their values could be estimated, true, or a mixture. For example, a PFM could output the probability of detecting a car for detection tasks or a pose distribution for localization tasks given source states such as camera parameters, atmospheric conditions like fog intensity and ambient lighting, and potential target cars. For either task, we