Self-supervised cross frame-rate event-frame depth learning without ground truth Authors: Arun Kumar Arigela, D. David Neels Ponkumar, Martin Margala, K. Jayasri, A. Leela Sravanthi, Prasun Chakrabarti Published: 2026 ABSTRACT Event-based cameras provide high-temporal-resolution motion information and wide dynamic range, making them suitable for depth estimation in challenging dynamic environments. However, existing depth learning approaches often rely on ground-truth supervision, handcrafted event representations, or computationally expensive fusion strategies, limiting scalability and real-time deployment. Recent self-supervised methods partially address annotation dependency but struggle with cross-frame-rate alignment, temporal instability, and inefficient fusion of event and frame modalities. To overcome these limitations, this study proposes a novel Self-Supervised Event–Frame Self-Attention Transformer (EF-SAT) framework for depth estimation without ground-truth depth supervision. The proposed method employs a Vision Transformer-based encoder to jointly model event streams and intensity frames, while a cross-frame-rate self-attention mechanism captures spatiotemporal dependencies across heterogeneous data rates. The framework further carries revolutionary self-sampling based primarily on bilinear interpolation, timescale-consistent intensity decoding, and ego-motion optimization that uses absolute trajectory error (ATE) and relative pose error (RPE) constraints to improve geometric balances in successive frames. Depth and pose are learned using photometric reconstruction, event–frame alignment, temporal consistency, and smoothness constraints in a fully self-supervised manner. Experiments conducted on the DAVIS-240 C event–frame dataset demonstrate that the proposed approach achieves superior performance, attaining an Absolute Relative Error of 0.114 and RMSE of 0.163, outperforming state-of-the-art deep learning baselines. Overall, the proposed EF-SAT model demonstrates robust, temporally stable, and accurate depth estimation in dynamic scenes, highlighting its potential for real-world applications such as robotics, autonomous navigation, and event-driven perception systems. Introduction The correct recognition of the structure of a three-dimensional scene is an essential requirement in computer vision tasks like autonomous navigation, robotics and augmented reality1. Conventional methods of depth estimation use frame-based cameras which record intensity images in constant frame rates. Nevertheless, these traditional sensors have automatic constraints in tough environments, such as high motion rate, low light, and an environment with sudden change of illumination2,3. Conversely, event cameras deliver asynchronous spatial-temporal data recording the change of pixel intensities in the resolution of microseconds4. The characteristic feature makes it possible to capture fast dynamic scenes with minimum motion blur and maximum temporal fidelity5. Combining data of event cameras with low-rate intensity images has become one of the most promising approaches to address the limitations of regular cameras, which can enable dense depth estimation, and maintain a temporal consistency between frames6. Although event-frame fusion has a potential, there are a number of challenges that remain in the existing methodologies7. Traditionally, early methods of depth estimation made use of supervised learning models which needed extensive ground-truth depth annotations, which are costly and time-intensive to acquire8. Recent study has explored into the self-supervised learning models to reduce the reliance on labelled datasets, using photometric consistency and cross-modal losses on alignment9. CNN-based systems have been widely used in the depth prediction of images and events, but have been known to have weaknesses in capturing spatial and temporal dependencies of long-range, especially in an asynchronous stream of events10. Models with self-attention mechanisms built on transformers offer a compelling alternative to model global correlations among multi-modality and multi-temporal data11. However, current transformer methods of depth estimation have been highly one-sided either in image sequence or event stream and have made no major use of cross-frame-rate fusion between an asynchronous event stream and a low-frame-rate image stream12. The gap shows that there is the need to have a unified model that is capable of acquiring feature-rich spatial-temporal representations using a combination of multimodal in a self-supervised fashion, and that can produce accurate depth estimates without the use of ground-truth depth labels. The proposed paper presents a self-supervised EF-SAT dense depth estimation, that is, a fusion between asynchronous streams of events and low-frame-rate intensity images. Through the cross-frame-rate self-attention, the framework is able to capture the long-range spatial-temporal dependence and therefore prediction of depth is obtained accurately even without ground-truth annotations. Recent innovation and its limits Recent depth estimation progress has investigated self-supervised architecture, event-camera fusion and transformer-based architecture. Event cameras deliver asynchronous data of high temporal resolution, which makes it possible to have motion-robust perception of dynamic scenes13. Self-supervised methods have minimized the use of labelled depth maps by using photometric consistency and cross-modal alignment. Transformer models have been shown to be superior in the capability of capturing the long-range dependencies when compared with convolutional networks. Nonetheless, the existing innovations have significant drawbacks. The current transformer-based approaches tend to treat each single modality separately, and make poor use of cross-modal information between events and frames14. Furthermore, the majority of methods are limited to regular frame rates, which prevents their use in asynchronous sensor data tasks. The limited feature fusion mechanisms and sparse representations of events are impediments to accurate depth reconstruction especially in complicated dynamic environments. These difficulties lead to the need to have a single framework which can cross-frame-rate fuse to achieve proper self-supervised depth learning. Problem statement The precise estimation of depth is also a basic issue in dynamic and complicated settings and the use of classical frame-based cameras. Traditional frame based techniques are affected by motion blur, poor light sensitivity and performance loss when there is a sudden change in illumination15. Even though event cameras have excellent temporal resolution and resistance to changes in light, due to their sparsity and asynchronous nature, direct depth information cannot be easily inferred. The current deep learning models are mostly either single-modality based or are based on CNN-based models which do not fully capture long-range spatial-temporal interactions between event-frame data16. Moreover, the majority of approaches assume equal frame rates and rely on supervised learning that requires expensive and hard to obtain ground-truth depth labels. These are scalability and practicality limitations. In order to overcome these issues, this study presents a self-supervised EF-SAT that is able to effectively merge asynchronous event streams and the low-frame-rate intensity images that can densely estimate depth without the need to use ground-truth depth labels. Research motivation The rationale behind the current research is the need to eliminate the shortcomings of the current depth estimation methods in dynamic and low-light conditions. With the combination of asynchronous event streams and low-frame rate intensity images, the proposed framework will help capture rich spatial-temporal representations. The use of self-supervised learning methods removes the need to use expensive ground-truth depth annotations, and the transformer-based self-attention allows effective cross-modal and time-scale fusion. The aim of this approach is to give correct and real-time depth information of robotics and autonomous navigation systems. Research significance This study makes a contribution to the field of depth estimation by proposing a single unified, self-supervised framework to combine event and frame information. The developed EF-SAT model overcomes the shortcomings of traditional CNN and single-modality transformer-based methods enabling the model to provide robust results through dynamic, low light, and high-speed conditions. It is able to remove the need to use ground-truth depth annotations, thereby lowering the cost of data acquisition and improving scalability. The method has a great potential in autonomous systems, robotics, and real-time vision application that needs precise and reliable depth perception. Key contributions Introduces a self-supervised Event–Frame Self-Attention Transformer for fusing asynchronous event streams with low-frame-rate intensity images for dense depth estimation.Cross-frame-rate fusion: Effectively models long-range spatial-temporal dependencies between events and frames using multi-head self-attention, overcoming limitations of CNN-based architectures.Eliminates the need for ground-truth depth annotations, leveraging photometric consistency, event-frame alignment, and temporal consistency losses.Demonstrates resilience to high-speed motion, low-light conditions, and abrupt illumination changes, outperforming existing single-modality methods.Designed for efficient inference, enabling potential deployment in robotics, autonomous navigation, and other real-time vision applications. Introduces a self-supervised Event–Frame Self-Attention Transformer for fusing asynchronous event streams with low-frame-rate intensity images for dense depth estimation. Introduces a self-supervised Event–Frame Self-Attention Transformer for fusing asynchronous event streams with low-frame-rate intensity images for dense depth estimation. Cross-frame-rate fusion: Effectively models long-range spatial-temporal dependencies between events and frames using multi-head self-attention, overcoming limitations of CNN-based architectures. Cross-frame-rate fusion: Effectively models long-range spatial-temporal dependencies between events and frames using multi-head self-attention, overcoming limitations of CNN-based architectures. Eliminates the need for ground-truth depth annotations, leveraging photometric consistency, event-frame alignment, and temporal consistency losses. Eliminates the need for ground-truth depth annotations, leveraging photometric consistency, event-frame alignment, and temporal consistency losses. Demonstrates resilience to high-speed motion, low-light conditions, and abrupt illumination changes, outperforming existing single-modality methods. Demonstrates resilience to high-speed motion, low-light conditions, and abrupt illumination changes, outperforming existing single-modality methods. Designed for efficient inference, enabling potential deployment in robotics, autonomous navigation, and other real-time vision applications. Designed for efficient inference, enabling potential deployment in robotics, autonomous navigation, and other real-time vision applications. The rest of this paper is organized as follows. Section “Related works” reviews relevant literature on depth estimation, event-frame fusion, and transformer-based methods. Section “Proposed event frame self attention transformer framework” presents the proposed EF-SAT methodology, including data preparation, feature encoding, and self-supervised learning. Section “Result and discussion” discusses experimental results and performance analysis. Finally, Sect. “Conclusion and future work” concludes the study and outlines directions for future work. Related works Event-based video reconstruction and representation learning Lu et al.17 suggest a self-supervised system to deal with rolling shutter artifacts and low frame rate with conventional cameras by reconstructing high-frame-rate video with event cameras. The aim of the study is to retrieve distortion-free slow-motion videos with minimum bandwidth without relying on large scale real-world supervised datasets. The authors present a representation of displacement fields that represents dense and non-linear spatiotemporal motions of pixels in case of exposure, with which rolling shutter and global shutter images can be mutually reconstructed. The framework combines rolling shutter- global shutter mapping with inverse global shutter- rolling shutter reconstruction and rolling shutter warping to obtain self-supervision. Experiments using four datasets and user experiments show that removing distortions are successful, bandwidth is reduced by 94% and inference can occur in 16 ms per frame. Nonetheless, its major limitation is that it has not been quantitatively validated on real-world data, with experiments mainly done on simulated data, which is a limitation to real-world applicability. Qu et al.16 introduce a self-supervised self-data-driven data-driven approach to enhance event-stream representations to event-based vision tasks that overcomes the drawbacks of noise-sensitive manual representations. The aim of the study is to improve the quality and strength of event-stream representation in order to be able to apply to the usual learning models more efficiently. The authors introduce EvRep, a spatiotemporal statistical representation, and a self-supervised representation generator, RepGen that leverages the theoretical link of asynchronous event-streams with synchronous APS frame to generate polished representations called EvRepSL. Comprehensive testing on event-based classification and optical flow datasets shows significant improvement in performances compared to the current techniques and good generalizations to various event cameras and tasks. Nevertheless, the main limitation is that the performance of noise reduction is limited by the challenge of acquiring correct event-wise ground truth in practice. Self-supervised event-based depth estimation To address the shortcomings of using expensive and time-intensive depth annotations, Zhu et al.18 suggest a self-supervised event-based monocular depth estimation that they refer to as EMoDepth. The idea of the study is to make proper depth estimation in high-speed and problematic light scenes when the regular cameras have a negative performance. The suggested approach takes advantage of cross-modal consistency between asynchronous events and pixel-aligned intensity frames during training and uses only events at inference time. Multiscale skip-connection architecture is an architecture that is meant to combine features and ensure high-inference speeds. The experiments that are carried out on the MVSEC and the DSEC datasets indicate that EMoDepth has a state of art performance that is superior to the supervised event-based and unsupervised frame-based approaches. Nevertheless, the major limitation is low utilization of dynamic scene information, since the management of moving objects and creation of smaller yet more informative event representations are issues. Xiao et al.15 introduce a multimodal monocular dense depth estimation system to address the drawbacks of frame cameras in abnormal light circumstances. Research is intended to enhance the accuracy of depth prediction methods by making good use of the complementary advantages of event cameras and traditional frame data. The presented approach utilizes the transformer-based design that comprises of a multimodal encoder and a joint decoder, in which self-attention networks learn local and global relationships between frame patches and event tensors, and then integrate multi-scale features to provide slow-scale depth reconstruction. The results of the experiments conducted with the Event Scape and MVSEC data sets indicate that the approach has the state-of-the-art performance, and it is significantly better than the existing one. The major disadvantage is though that compared to transformer-based multimodal fusion, there is a higher computational complexity that can restrict its application in real time in resource-constrained systems. Zhang et al.19 suggest a depth estimation system that is energy-efficient to deal with the issue of event-camera data, such as non-standard spiking outputs and small labelled datasets. The objective of the study is to obtain correct depth estimation with low computational and energy expenses to resource-limited systems. To overcome data scarcity, the authors present a Spike-Driven Transformer Network (SDT) that has spike-based attention and residual mechanisms, a multi-stage fusion depth head, and cross-modality knowledge distillation based on a pre-trained vision foundation model. Experiment results on both synthetic and real-world event data have demonstrated strong performance improvement, such as a 49% decrease in absolute relative error, 39.77% decrease in squared relative error, 70.2% decrease in energy usage and 42.4% decrease in model parameters. Nevertheless, there is a potential limitation of validation being limited to a smaller set of datasets and platforms, and the implementation on specialized spiking devices and larger real-world validation remains not implemented. Duan et al.20 discuss the drawbacks of monocular depth estimation in dynamic scenes and difficult lighting environments by capitalizing on the defining advantages of event cameras and traditional frame cameras. The research objective is to enhance the accuracy and sensitivity of depth estimation by using good multimodal fusion. The authors introduce a new Coordinate Attention Gated Recurrent Unit (CAGRU), comprising of coordinate attention as an attention gate with a traditional convolutional gate to learn spatial, temporal, and inter-channel dependencies altogether. The design increases the information density of sparse event data, and allows effective frame information fusion with features. Experimental tests on various publicly available data sets show that the performance is significantly better than the current methods. Nevertheless, the major limitation is the use of synthetic data, which does not perfectly match the real-world data distributions and can cause bias and restrict generalization in the context of practical applications. Event–frame calibration and motion analysis Hu et al.21 examine the calibration of event-frame stereo camera systems to optimize on the complementary benefits of event cameras and traditional frame cameras. The objective of the research is to obtain the precise and real time dynamic calibration of such hybrid stereo configurations. The authors suggest a dynamic calibration scheme that jointly uses circle-grid pattern recognition, a sliding-window based time matching approach to pairing events with frames and an event warping and synchronization algorithm based on patch-size motion compensation, is computationally efficient. A pose- landmark graph is thereafter formed to optimize global pose between two cameras. The experimental findings on self-recorded datasets indicate that it has high calibration accuracy, good real-time performance and can be deployed in a practical manner. Nevertheless, the method has less robustness in difficult environments like the presence of strongly noisy scenes, overloaded events or even non parallel planes of pixels between the two cameras, and this can impact the calibration stability and accuracy. By using bio-inspired event cameras, Zhao et al.22 solve the problems with motion of objects and tracking them in rapidly moving scenarios and with changes in illumination. The goal of the study is to enhance the strength of detection, continuity of tracking, and range estimation of dynamic scenes. The authors suggest a mixed framework, which is the combination of event frames and standard frames as the case of object detection with probabilistic and colour cues, then a detection-based tracking algorithm with event frames and better kernel correlation filter to minimize missed detections. Also, another distance estimation technique which uses event-frame tracking and the similar triangle theory is presented. The experimental results prove the successful moving object detection, dependable track performance, and the distance measurement accuracy. Nevertheless, one of the major weaknesses is the quality of event-frame filtering and the capability to resist any complex changes in illumination because noise and adverse lighting conditions may deteriorate detection and tracking abilities. Transformer-based spatial–temporal modelling To overcome the temporal inconsistency and feature misalignment problem in the case of fast moving aerial motion, Yongsheng et al.23, proposed SiamCTCA for UAV tracking. To enhance spatial–temporal feature correspondence between consecutive frames, the study adopted a Cross-Temporal Correlation Aggregation mechanism embedded into a Siamese network, based on the UAV tracking benchmark datasets. The architecture improved the robustness to occlusion, and enhanced motion-aware feature learning for robustness under fast viewpoint variation. The results of experimental evaluation showed that the tracker has high tracking precision and temporal stability than the traditional Siamese tracker. Similarly, Wang et al.24, proposed a multi-channel complex residual network with adaptive sphere-space decision boundaries for embedded RF fingerprint interpretation. The technique used involved complex residual feature extraction and adaptive boundary optimization for increasing the reliability of classification in noisy environments. The experimental results showed the identification accuracy improvement and feature discrimination ability. However, the main tasks of the aforementioned studies were focusing on tracking and/or signal interpretation rather than geometric depth inference, which restricts the direct applicability to self-supervised event-frame depth estimation. Table 1Summary of existing studies.ReferencesMethodAdvantagesLimitationsLu et al.17Self-supervised RS-to-GS video reconstruction using event cameras and displacement field modellingRemoves rolling shutter distortion, enables high-frame-rate slow-motion video, achieves large bandwidth reduction and fast inferenceLacks quantitative validation on real-world datasets; evaluation mainly on simulated dataQu et al.16Self-supervised event-stream representation learning (EvRepSL) using APS–event correlationImproves representation quality, task- and camera-agnostic, outperforms handcrafted representationsLimited noise suppression due to difficulty in obtaining accurate event-wise ground truthZhu et al.18EMoDepth: self-supervised event-based monocular depth estimation using cross-modal consistencyEliminates need for depth annotations, high inference speed, state-of-the-art depth accuracyLimited handling of dynamic objects and dependence on event representation qualityXiao et al.15Transformer-based multimodal depth estimation with event–frame fusionCaptures global and local spatiotemporal dependencies, achieves superior depth accuracyHigh computational complexity, challenging for real-time or edge deploymentZhang et al.19Spike-driven transformer network (SDT) with knowledge distillation for depth estimationEnergy-efficient, reduced model size, strong depth accuracy with limited dataValidation limited to selected datasets and platforms; deployment on spiking hardware not yet demonstratedDuan et al.20Coordinate attention gated recurrent unit (CAGRU) for event–frame fusionEnhances spatial–temporal–channel feature modeling, improves robustness and accuracyReliance on synthetic datasets may limit generalization to real-world scenariosHu et al.21Dynamic calibration framework for event–frame stereo camera systemsHigh real-time performance, accurate synchronization, easy deploymentSensitive to noisy or overloaded events and non-parallel camera configurationsZhao et al.22Event-frame-based object detection, tracking, and distance estimationRobust to fast motion, reduced missed detections, effective distance measurementPerformance degrades under complex illumination and noisy event-frame conditionsYongsheng et al.23SiamCTCA: Cross-temporal correlation aggregation siamese network for UAV trackingImproves temporal correlation learning, robust under fast motion and occlusion, enhances spatial–temporal feature aggregationPrimarily designed for tracking tasks rather than geometric depth estimationWang et al.24Multi-channel complex residual network with adaptive sphere-space decision boundariesEnhances feature discrimination, improves classification robustness under noisy environmentsFocused on RF signal interpretation rather than event-frame geometric learning Table 1Summary of existing studies.ReferencesMethodAdvantagesLimitationsLu et al.17Self-supervised RS-to-GS video reconstruction using event cameras and displacement field modellingRemoves rolling shutter distortion, enables high-frame-rate slow-motion video, achieves large bandwidth reduction and fast inferenceLacks quantitative validation on real-world datasets; evaluation mainly on simulated dataQu et al.16Self-supervised event-stream representation learning (EvRepSL) using APS–event correlationImproves representation quality, task- and camera-agnostic, outperforms handcrafted representationsLimited noise suppression due to difficulty in obtaining accurate event-wise ground truthZhu et al.18EMoDepth: self-supervised event-based monocular depth estimation using cross-modal consistencyEliminates need for depth annotations, high inference speed, state-of-the-art depth accuracyLimited handling of dynamic objects and dependence on event representation qualityXiao et al.15Transformer-based multimodal depth estimation with event–frame fusionCaptures global and local spatiotemporal dependencies, achieves superior depth accuracyHigh computational complexity, challenging for real-time or edge deploymentZhang et al.19Spike-driven transformer network (SDT) with knowledge distillation for depth estimationEnergy-efficient, reduced model size, strong depth accuracy with limited dataValidation limited to selected datasets and platforms; deployment on spiking hardware not yet demonstratedDuan et al.20Coordinate attention gated recurrent unit (CAGRU) for event–frame fusionEnhances spatial–temporal–channel feature modeling, improves robustness and accuracyReliance on synthetic datasets may limit generalization to real-world scenariosHu et al.21Dynamic calibration framework for event–frame stereo camera systemsHigh real-time performance, accurate synchronization, easy deploymentSensitive to noisy or overloaded events and non-parallel camera configurationsZhao et al.22Event-frame-based object detection, tracking, and distance estimationRobust to fast motion, reduced missed detections, effective distance measurementPerformance degrades under complex illumination and noisy event-frame conditionsYongsheng et al.23SiamCTCA: Cross-temporal correlation aggregation siamese network for UAV trackingImproves temporal correlation learning, robust under fast motion and occlusion, enhances spatial–temporal feature aggregationPrimarily designed for tracking tasks rather than geometric depth estimationWang et al.24Multi-channel complex residual network with adaptive sphere-space decision boundariesEnhances feature discrimination, improves classification robustness under noisy environmentsFocused on RF signal interpretation rather than event-frame geometric learning Summary of existing studies. Recent progress in event-based vision, multimodal fusion, Transformer learning, and spatial–temporal attention modelling is summarized in Table 1. The advantages of existing studies were shown in motion analysis, representation learning, robustness of tracking, and depth estimation accuracy. But there are restrictions in its computational complexity, dynamic-scene handling, cross-frame rates modelling and real-world generalization. The significance of temporal correlation aggregation and feature specialization is further emphasised by attention-based studies which have been introduced recently such as SiamCTCA and adaptive residual learning. To address these research challenges, the proposed EF-SAT framework proposes cross-frame-rate self-attention fusion to jointly capture spatial geometry and asynchronous motion information and enhances temporal consistency, computational efficiency, and self-supervised depth estimation accuracy. Proposed event frame self attention transformer framework The proposed approach introduces a unified self-supervised system of depth estimation which builds upon the complementary nature of event-based and frame-based vision sensors. The system is trained to be depth-agnostic using time-varying information on geometry and consistency of appearances. With synchronized intensity frames and event streams, which have high temporal resolution, as inputs, the framework performs the modality specific preprocessing to provide spatial temporal alignment. Both modalities are converted to patch-based token representations which are augmented with space and time positional encodings. These tokens are further fed into the EF-SAT which is the main learning unit. In EF-SAT, cross frame rate and self-attention on latent features allow the effective combination of frame-based spatial structure with asynchronous information on events. The resulting combined representation of latent represents both scene geometry and motion dynamics. The next step is parallel decoding branches that forecast dense depth maps and relative camera motion that are employed to impose self-supervised learning constraints by view synthesis and cross-modal consistency. In inference, the trained system simply allows generating depth maps on unseen data without any other supervision. This design is end-to-end allowing it to be robust during high velocity motion and adverse illumination, but also be scaled to real-time deployment. The workflow of the proposed framework is illustrated in Fig. 1. Fig. 1Workflow of the proposed EF-SAT framework. Workflow of the proposed EF-SAT framework. Data collection This study utilizes publicly available event-based vision datasets, which are compiled in the event-based-camera-dataset repository available in GitHub, which contains several benchmark datasets recorded with DAVIS event cameras [25]. These datasets offer synchronized asynchronous event streams and low-frame-rate intensity images, which can be successfully fused with events to produce depth learning. The data set is gathered in a wide range of interior and external conditions, including fast motion and dynamic scenes in combination with different levels of light. Streams of events can document the pixel-level changes of brightness at a microsecond time resolution whereas traditional grayscale pictures can provide complementary structural data. Some sequences also provide ground-truth depth or camera pose, not used to be supervised, but to be evaluated, and this also falls within the paradigm of self-supervised learning followed in this paper. The variability of the motion patterns, geometry of the scene, and light variations in the dataset makes the process of training robust and thorough validation of the suggested Event Frame Self-Attention Transformer in conditions of real scenarios. Data preprocessing Effective fusion of the stream of events and intensity frames, preprocessing is necessary to deal with the inherent disparities in temporal resolution, data density, and representation of these two streams. Before the encoding of features and the fusion through transformers, the raw sensor values are converted into aligned and noise-reduced and structurally compatible values. The preprocessing stage provides the temporal coherence between asynchronous and synchronous frames as well as the high-frequency motion information of event cameras. The subsequent subsections outline the preprocessing pipeline, comprising event stream normalization, temporal segmentation, event representation construction, frame preparation, and cross-modal alignment, that find applications in all the phases of enabling stable and meaningful self-supervised learning in the proposed EF-SAT Transformer framework. Event representation construction In order to learn with transformer architectures, the events are scarified into a dense representation of a tensor. Any set of segmented events is coded into a voxel grid that maintains the location of space, the order of time, and the polarity. The values of each voxel are calculated as in Eq. (1).1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:V(x,y,b)=\sum\:_{{e}_{i}\in\:{\mathcal{E}}_{k}}{p}_{i}\cdot\:\delta\:(x-{x}_{i},y-{y}_{i},b-{b}_{i})$$\end{document} where V(x,y,b) is the event voxel grid value at spatial location \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:(\mathrm{x},\mathrm{y})$$\end{document} and temporal bin b, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{p}_{i}$$\end{document} is the polarity of event \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{e}_{i}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{b}_{i}$$\end{document} is the temporal bin index of event \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{e}_{i}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{x}_{i},\:{y}_{i}$$\end{document} is the pixel location of event \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{e}_{i}$$\end{document}, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\delta\:(\cdot\:)$$\end{document} is the Kronecker delta. This representation maintains polarity of events and time ordering allowing the model to learn depth cues which are motion aware. Frame normalization and spatial alignment The intensity frames are normalized to minimize the fluctuations in the illumination and to achieve the numerical stability in training. The step normalizes the pixel values within the data set. Frames that are received by the DAVIS sensor are converted to grayscale and normalised to achieve numerical stability in the training process. Each frame \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{I}_{k}\in\:{\mathbb{R}}^{H\times\:W}$$\end{document}is normalized as in Eq. (2).2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\widehat{I}}_{k}=\frac{{I}_{k}-{\mu\:}_{I}}{{\sigma\:}_{I}}$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mu\:}_{I}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\sigma\:}_{I}$$\end{document} denote the mean and standard deviation of pixel intensities computed over the dataset. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\widehat{I}}_{k}$$\end{document} is the normalized intensity frame, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{I}_{k}$$\end{document} is the raw intensity frame at time \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{T}_{k}$$\end{document}. Then the modalities are correspondently pixel-wise matched by resizing the frames to the spatial resolution of the event voxel grid. This spatial alignment is necessary to have effective cross-modal attention wherein the transformer is able to match the motion events with the image structures. There is no geometric augmentation to prevent interfering with time continuity between successive frames. Cross-modal temporal alignment Event timestamps are centred on the frame time to reduce the temporal misalignment of events and frames to provide the correct cross-modal correspondence of events and frames. Although the temporal segmentation is undertaken, there might be some minor misalignments because of sensor latency and asynchronous sampling. The temporal offset is determined as in Eq. (3).3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\Delta\:}t=\frac{1}{\mid\:{\mathcal{E}}_{k}\mid\:}\sum\:_{{e}_{i}\in\:{\mathcal{E}}_{k}}({t}_{i}-{T}_{k})$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\Delta\:}{t}_{k}$$\end{document} is the Mean temporal offset of events in segment k, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mid\:{\mathcal{E}}_{k}\mid\:$$\end{document} is the number of events in segment k, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{t}_{i}$$\end{document} is the time stamp of event \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{e}_{i}$$\end{document}, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{T}_{k}$$\end{document} is the time stamp of the corresponding intensity frame. This correspondence guarantees that the information regarding the motion of the events does relate correctly to the visual structures in the frame. This synchronization is essential to provide self-supervised photometric and event-frame consistency losses which have a direct effect to the consistency and precision of depth estimation in dynamic scenes. Event–frame input tokenization for EF-SAT-based depth learning Input tokenization in the current study is designed to transforming heterogeneous visual representations such as synchronous intensity images and asynchronous event streams into a single sequence of tokens which can be fed into the transformer. In contrast to CNN pipelines based on local receptive fields, EF-SAT framework suggests tokenized inputs which maintain spatial structure, time motions, and cross-modal correspondence. This section explains the construction and segmentation of frame-event pairs into tokens and the embedding of this into the frame-event pairs to directly compute self-supervised depth estimation by means of the cross-frame-rate attention. For each intensity frame captured at time \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{T}_{k}$$\end{document}, the corresponding event information consists of all events occurring between the current frame and the next frame. This pairing allows the model to observe scene motion that occurs between low-frame-rate images, which is essential for depth inference in dynamic environments. It is expressed as in Eq. (4).4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathcal{P}}_{k}=\left({I}_{k},{\mathcal{E}}_{k}\right)$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathcal{P}}_{k}$$\end{document} is the Frame–event pair at index k, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{I}_{k}$$\end{document} is the intensity frame captured at timestamp \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{T}_{k}$$\end{document}, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathcal{E}}_{k}$$\end{document} is the Set of events occurring within \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:[{T}_{k},{T}_{k+1})$$\end{document}. This combination allows the EF-SAT model to reason together concerning the static appearance (frames) and dynamic motion (events), which is important in obtaining precise depth estimates without support of ground-truth. Asynchronous event streams are stored in voxel grids, but the temporal ordering and the intensity of the motion of the events within the temporal structure of each voxel bin is kept. This discretization approach minimizes sparsity of events and instability of the computation for such motion and retains the essential high frequency-based motion dynamics needed for the cross-frame-rate alignment. In very fast movements, minor temporal smoothing may be observed, but in the voxel representation, stable attention learning in the temporal space is achieved and efficient multimodal interaction between event and frame tokens is realized. Both the normalized intensity frame and the equivalent event voxel grid are broken in non-overlapping spatial patches. The patches can model the association between distant regions using self-attention instead of local convolution since each patch represents a local spatial region. It is computed as in Eq. (5).5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{I}_{k}\to\:\left\{{P}_{k}^{f}\right(i){\}}_{i=1}^{N},{V}_{k}\to\:\{{P}_{k}^{e}\left(i\right){\}}_{i=1}^{N}$$\end{document} Here \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{P}_{k}^{f}\left(i\right)$$\end{document} is the i-th frame patch from frame \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{I}_{k}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{P}_{k}^{e}\left(i\right)$$\end{document} is the i-th event patch from voxel grid \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{V}_{k}$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:N=\frac{H\times\:W}{{p}^{2}}$$\end{document} is the total number of patches, H,W are the Height and width of the input, and p is the Patch size (e.g., \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:16\times\:16$$\end{document}). Spatial correspondence guarantees by the use of identical patch partitions a correspondence between frame and event tokens, necessary to effective cross-modal self-attention when estimating depth. The frame patch as well as the event patch are flattened and mapped into a fixed dimension embedding space via learnable linear projections. This converts heterogeneous visual data to modality consistent token vectors. It is expressed as in Eq. (6).6\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathrm{z}}_{i}^{f}={\mathrm{W}}_{f}\cdot\:\mathrm{vec}\left({P}_{k}^{f}\right(i\left)\right),{\mathrm{z}}_{i}^{e}={\mathrm{W}}_{e}\cdot\:\mathrm{vec}\left({P}_{k}^{e}\right(i\left)\right)$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathbf{z}}_{i}^{f}$$\end{document} is the Embedded frame token, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathbf{z}}_{i}^{e}$$\end{document} is the Embedded event token, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathrm{W}}_{f},{\mathrm{W}}_{e}$$\end{document} is the learnable projection matrices, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mathrm{vec}(\cdot\:)$$\end{document} is the flattening operation. Separate embedding matrices enable the model to maintain modality specific attributes and this is done by projecting the two modalities onto a common transformer compatible space. Because transformer architectures are not spatially aware, positional encodings are introduced to every token to store spatial layout information in patches. It is computed as in Eq. (7).7\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\stackrel{\sim}{\mathbf{z}}}_{i}^{m}={\mathbf{z}}_{i}^{m}+{\mathrm{p}}_{i}$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\stackrel{\sim}{\mathbf{z}}}_{i}^{m}$$\end{document} is the position-aware token for modality \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:m\in\:\{f,e\}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathrm{p}}_{i}$$\end{document} is the learnable positional embedding for patch i. Positional encoding allows the EF-SAT model to plan relative spatial geometry that plays a crucial role in learning the depth structure of a monocular view. The last input to EF-SAT layer is built by using frame tokens and event tokens combined into one sequence, which allows cross-modal interaction via self-attention. The sequence of unified tokens enables the EF-SAT transformer to explicitly model spatial-temporal relations between appearance and motion which is the direct impetus in self-supervised depth learning. Event–frame self-attention transformer layer The main learning element of the proposed self-supervised depth estimation system is the EF-SAT. It is mainly designed to integrating effectively the complementary information of two basic sensing modalities; namely low-frame-rate intensity images which code the rich spatial appearance and scene structure, and high-temporal-resolution event streams that encode the fine-grained motion dynamics. In contrast to traditional convolutional networks, which use local receptive fields and find it challenging to define long distance dependencies, EF-SAT layer uses self-attention to define more global spatial-temporal relationships between event and frame representations. This feature is necessary especially in-depth estimation without ground-truth guidance where the geometric knowledge is indirectly learned by examining motion and appearance consistency. The EF-SAT layer makes cross-frame-rate interaction modelling, joint encoding of modality-specific cues and the generation of unified latent representations, which underlie the basis of the proposed self-supervised learning strategy, possible, and facilitates effective depth and pose learning by view synthesis and time constraints. The EF-SAT architecture is designed not just to enhance the learning of spatial–temporal features, but also to be computationally efficient. The proposed cross-frame-rate attention mechanism reduces unnecessary attention computations by selectively focusing on the event regions where motion information is important and the frame regions where spatial information is important. Besides, light-weight decoding layers and shared latent feature representations help to reduce parameter complexity and floating-point operations (FLOPs), maintaining the geometric learning capacity while allowing to improve the performance of real-time inference. The architecture of the vision transformer is illustrated in Fig. 2. Fig. 2Vision transformer architecture. Vision transformer architecture. Event–frame latent feature alignment The main role of aligning heterogeneous sensory data into unified latent representation that can be used in cross-frame-rate reasoning is played in the proposed self-supervised event-frame depth learning framework, where the EF-SAT encoder acts as the primary element. The tokenized intensity frames and event-based representations in this stage have wide differences in terms of temporal density, spatial sparsity, and information distribution in the model. Whereas intensity frames capture mostly the static appearance and the geometry of a scene, event streams encode the high-frequency motion-related information of capturing the slight alterations of brightness due to camera or object motion. The explicit encoding approach would result in representational imbalance and inefficient interaction of attention were there to be direct fusion of these modalities. The EF-SAT encoder overcomes this difficulty by mapping both modalities into the same embedding space retaining the complementary nature of modalities. Formally every token is coded according to Eq. (8).8\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathrm{h}}_{i}^{m}={\mathrm{W}}_{e}^{m}{\mathrm{z}}_{i}^{m}+{\mathrm{p}}_{i}^{s}+{\mathrm{p}}_{k}^{t},m\in\:\{f,e\}$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathrm{h}}_{i}^{m}$$\end{document} denotes the encoded latent feature of the i-th token from modality m, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathrm{z}}_{i}^{m}$$\end{document} represents the corresponding input token embedding, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathrm{W}}_{e}^{m}$$\end{document} is a modality-specific learnable projection matrix that maps raw embeddings into a common latent dimension. The term \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathrm{p}}_{i}^{s}$$\end{document}encodes spatial positional information associated with the patch location, preserving geometric structure necessary for depth reasoning, while \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathrm{p}}_{k}^{t}$$\end{document} represents temporal positional encoding that identifies the frame interval from which the token originates. This temporal representation is essential in the differentiation of changes due to motion and changes due to appearance at rest which is a major requirement of self-supervised depth estimation. To address this problem, the EF-SAT encoder uses temporal positional encoding to put the temporal interval information directly into the EF-SAT encoder latent representation, helping to clarify low-frame-rate frame tokens from high-frequency event token s. This mechanism blocks the ambiguity of features from different modalities, so the transformer can maintain the modality-specific temporal properties rather than averaging interactions between different heterogeneous inputs. By such an encoding scheme frame tokens add both global structural and photometric information, but event tokens add movement information corresponding to scene dynamics between successive frames. Using both modalities together, in a common latent space that is enriched with spatial and time context, the EF-SAT encoder makes the following self-attention process possible to create meaningful maps between appearance and motion. This is especially significant in the depth learning via motion parallax because closer objects are likely to produce a more intense and quicker event responses than far-off areas. The encoded features implicitly encode depth relations between features therefore do not use explicit depth annotations. In the framework of the present research, the EF-SAT encoder is essential in stabilizing self-supervised learning. The richer latent representations that were produced by the encoder enable the model to predict camera motion and scene geometry across time that directly enables the calculation of photometric and event-alignment losses that are utilized during training. The EF-SAT encoder provides a solid base to proper depth estimation in extreme situations, like rapid motion and low-illumination, where the conventional frame-based systems tend to perform poorly due to the joint embedding of spatial layout, temporal dynamics, and information of a particular modality. Cross-frame-rate spatial–temporal dependency modelling The cross-frame-rate self-attention mechanism is the major part of the proposed structure that will integrate both the spatial appearance information gained on intensity frames and the high-temporal-resolution motion information gained on event streams. The novelty of EF-SAT is that it can be used to attend simultaneously frame and event tokens, although the two have different temporal properties. The EF-SAT attention in contrast to traditional methods of attention where the attention mechanisms are based on homogeneous stimuli is specifically created to address the intrinsic imbalance between the temporal and representational properties of low-frame-rate images compared to asynchronous events. The attention mechanism in this study allows the model to selectively match event responses caused by motion to the corresponding spatial locations in the frame, which allows the model to infer depth in a self-supervised fashion and using motion parallax. Multi-head self-attention makes the model correlate the motion cues generated by the event with the corresponding regions of the image, which enables the model to infer depth when their frame information is ambiguous. Once the feature encoding is done, frame and event tokens are joined together, forming a single sequence, which the attention mechanism can then process together. Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mathrm{H}\in\:{\mathbb{R}}^{N\times\:d}$$\end{document}denote the combined latent representation, where Nis the total number of tokens from both modalities and d is the embedding dimension. Attention operation calculates the pairwise correlation of all tokens mathematically expressed in the form of Eq. (9).9\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathrm{Attention}}({\mathrm{Q}},{\mathrm{K}},{\mathrm{V}}) = {\mathrm{softmax}}\left( {\frac{{{\mathrm{QK}}^{\top} }}{{\sqrt {{\mathrm{d}}_{{\mathrm{k}}} } }}} \right){\mathrm{V}}$$\end{document} where query, key and value matrices are obtained by means of linear projections are denoted in Eq. (10).10\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mathrm{Q}=\mathrm{H}{\mathrm{W}}_{Q},\mathrm{K}=\mathrm{H}{\mathrm{W}}_{K},\mathrm{V}=\mathrm{H}{\mathrm{W}}_{V}$$\end{document} Here, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathrm{W}}_{Q},{\mathrm{W}}_{K},{\mathrm{W}}_{V}\in\:{\mathbb{R}}^{d\times\:{d}_{k}}$$\end{document} are learnable projection matrices, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{d}_{k}$$\end{document} denotes the dimensionality of the key space used for normalization. The softmax normalization is used to make sure that the norms of attention represent the relative significance of interactions between tokens. In the framework of the current inquiry, frame tokens are generally powerful query anchors because of the dense area coverage whereas event tokens are discriminatory key-value data that are aligned with the level of movement intensity and direction. The regions that show high motion produce more event responses resulting in a larger weighting of attention when relating to spatially compatible frame tokens. Consequently, the attention system gets to acquire implicitly to highlight foreground objects and depth-discontinuous areas which have more motion signals, and inhibit motionless or remote background areas. This behaviour is also highly beneficial to depth learning that is not trained using ground-truth, since it enables the model to learn relative depth ordering by the magnitude of motion and spatial correspondence. Multi-head attention is also used to increase the representational capacity so that the model can learn different interaction patterns in different subspaces. In the case of h attention heads, the result is calculated as in Eq. (11).11\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mathrm{A}=\mathrm{Concat}\left({\mathrm{A}}_{1},\dots\:,{\mathrm{A}}_{h}\right){\mathrm{W}}_{O}$$\end{document} where each head \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathrm{A}}_{j}$$\end{document} performs attention independently, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathrm{W}}_{O}$$\end{document} is a learnable output projection matrix. Such a design enables various heads to specialize in short-range space correspondences, long-range geometrical relations and temporal motion dependencies, which are combined to enhance performance on depth estimation. The multi-head attention strategy also gains from the fact that modality-wise interactives are multiplied on lower dimensional subspaces, which reduces the computational complexity of high dimensional global attention. Each of the different attention heads is learning a particular interaction pattern between event and frame tokens, based on the magnitude and temporal changes of the motion. Some heads achieve a fast changeover in the high motion areas, but others focus on maintaining frame consistency in geometric areas. This specialization enhances cross-frame-rate alignment and will not average out diverse modalities in the process of spatial-temporal feature fusion. The output of attention is further refined with the help of residual connections and feed forward network, resulting in the final fused representation as given in Eq. (12).12\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mathrm{U}=\mathrm{FFN}\left(\mathrm{A}\right)+\mathrm{H}$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mathrm{U}$$\end{document} denotes the unified latent feature map that integrates appearance, motion, and temporal context. The depth and pose decoders use this representation directly in the following processing of the framework. The cross-frame-rate self-attention mechanism is considered the key to the robust self-supervised learning in this study. The EF-SAT attention enables the network to make reasoning about geometry of the scene in a case of high-rate movement, low-light conditions and a non-textured surface by explicitly modelling asynchronous event interaction with synchronous frames. As a result, depth predictions based on this fused representation are more spatially consistent and temporally stable, which shows that the proposed attention strategy is effective in event frame depth estimation in the real world. The event tokens help to channel the attention to areas that are moving as the frame tokens help to give appearance context. Through this interaction, the model can learn relative depth using motion parallax without depth control monitoring. Geometry-aware depth and motion inference Depth and pose decoding are the last step of prediction in the proposed EF-SAT-based framework, which makes self-supervised learning possible without ground-truth depth maps. The latent representation of the cross-frame-rate self-attention layer is unified and contains the spatial geometry and the motion information of an event represented through event-frame fusion. Nevertheless, this latent representation does not directly give the supervisory signals. Rather, depth and pose decoders convert the latent features to physically meaningful representations dense depth maps and relative camera motion and can then be used to generate novel views, and provide geometric consistency between timestamps. The depth decoder is developed to predict a rich depth map of every single intensity frame through the gradual transformation of the unified latent features into pixel-wise depth estimations. Given the unified latent representation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathrm{U}}_{t}$$\end{document}corresponding to time step t, the depth prediction is formulated as in Eq. (13).13\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{D}_{t}={f}_{d}\left({\mathrm{U}}_{t}\right)$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{D}_{t}\in\:{\mathbb{R}}^{H\times\:W}$$\end{document} denotes the predicted depth map, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{f}_{d}(\cdot\:)$$\end{document} represents the depth decoding function. In this study, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{f}_{d}(\cdot\:)$$\end{document} consists of a lightweight decoding network with progressive up sampling layers that restore spatial resolution while preserving the geometric relationships encoded in the latent features. A bilinear interpolation and convolutional up-sampling layers produce progressive up-sampling without checkerboard noise that is typically seen with transposed convolution. The convolutional refinement further improves the boundary preservation and geometric continuity, allowing for efficient depth reconstruction for real-time event-frame depth estimation. The decoder combines the appearance information provided by frame tokens and the motion-sensitive one provided by event tokens in such a way that the depth can be inferred in the areas of texture lessness or poor illumination conditions. Significantly, the scale-consistent prediction of depth values is done across sequential frames, which is critical to imposing temporal coherence when training using self-supervision. The decoder uses common latent representations and motion-aware feature alignment across successive frames to minimize the scale drift between neighboring frames during depth decoding and ensure accurate geometric consistency across temporal sequences, thereby preserving the temporal scale consistency. Furthermore, decoder parameters are also shared across frames to enforce temporal consistency and photometric reconstruction constraints to promote consistent depth scaling between consecutive frames. This mechanism ensures that the scale of the predictions remains as stable as possible from one frame in time to the next and enhances the geometric goodness of self-supervised depth optimization in dynamic scene reconstruction. In parallel to the depth estimation, the pose decoding is in charge of estimating the relative motion of a camera between two consecutive frames. It is an essential part of self-supervised depth learning because camera motion gives it the geometric connection to compare depth predictions over time. The pose decoder is fed with the latent representations of two-time steps, and it predicts the relative rigid-body transformation as given in Eq. (14).14\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathrm{T}}_{t\to\:t+1}={f}_{p}\left({\mathrm{U}}_{t},{\mathrm{U}}_{t+1}\right)$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathrm{T}}_{t\to\:t+1}\in\:SE\left(3\right)$$\end{document} denotes the relative camera pose between frames t and t+1, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{f}_{p}(\cdot\:)$$\end{document} is the pose regression network. The transformation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathrm{T}}_{t\to\:t+1}$$\end{document} consists of a rotation matrix and a translation vector that jointly describe camera motion in three-dimensional space. This study uses the pose decoder to utilize motion information that is gathered by event tokens in order to enhance robustness in high motion conditions, where the conventional frame-based pose estimator tends to be unreliable. The depth prediction and pose are combined in order to reconstruct a desired frame with the help of a source frame through differentiable warping of views. Given the predicted depth \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{D}_{t}$$\end{document} and pose \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathrm{T}}_{t\to\:t+1}$$\end{document}, pixels from frame \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{I}_{t+1}$$\end{document} are projected into the coordinate frame of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{I}_{t}$$\end{document}, enabling reconstruction of the target image. The major self-supervised learning signal is photometric discrepancies between the reconstructed and original frame. This process enables the model to improve the quality of depth and pose estimates by minimizing rebuilding error, and this does not need any ground-truth depth labels. In the framework of this research, depth and pose joint estimation is the key of effective utilization of event frame data. Frames give spatial information that can be used to construct depth, whereas events give accurate temporal motion data which levels off pose estimation. Through the combination of these predictions, the suggested framework will guarantee that the learned depth maps become geometrically consistent through time as well as matched with the observed motion patterns. It leads to the trustworthy depth learning under difficult real-world conditions, such as high-speed motion and low-light conditions, which meets the essential goal of the self-supervised cross-frame-rate event-frame depth estimation. To quantitatively demonstrate the effectiveness of joint depth-pose learning, accuracy of the ego-motion estimation is assessed based on Absolute Trajectory Error (ATE) and Relative Pose Error (RPE). The ATE metric computes the overall deviation of the global trajectory of the predicted camera motion from the reference camera motion, while the RPE computes the local deviation of the camera motion throughout the frames. The lower the ATE and the RPE values, the more consistent are the geometries and the more reliable the pose estimation is in self-supervised optimization. These metrics are used to ensure the stability of the proposed depth–pose closed loop learning under dynamic motion. Self-supervised constraint for event–frame depth learning This study employs self-supervised learning is enabled through a series of geometry- and motion-consistent constraints which permit estimating depth without ground-truth guidance. The Event Frame Self-Attention Transformer predicts depth, relative camera motion which is tested by enforcing consistency between reconstructed intensity frames, the event activity and temporal depth behaviour. Such limitations are specifically tailored to capitalize on the complementary nature of frame-based appearance cues and event-based motion cues and provide strong depth learning in dynamic and brightness-challenging conditions. The main signal of supervision is based on the consistency of photometric reconstruction. With the estimated depth map and camera movement, a neighbouring frame is geometrically mapped into the reference view, and the error in reconstruction is reduced as in Eq. (15).15\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathcal{L}}_{photo}=\sum\:_{p}{\parallel\:{I}_{t}\left(p\right)-{\widehat{I}}_{t}\left(p\right)\parallel\:}_{1}$$\end{document} Here, p denotes a pixel location in the image plane; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{I}_{t}\left(p\right)$$\end{document} represents the observed pixel intensity at location p in the reference frame captured at time t; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\widehat{I}}_{t}\left(p\right)$$\end{document} denotes the reconstructed pixel intensity obtained by warping the neighbouring frame using the predicted depth \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{D}_{t}$$\end{document} and relative camera pose \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathbf{T}}_{t\to\:t+1}$$\end{document}. This loss imposes the property that the modelled depth accounts for pixel-wise intensity correspondences due to camera motion, which is the main self-supervisory signal of the current research. In order to explicitly add motion-sensitive event information, an event-frame alignment constraint is added. Because event streams encode asynchronous changes in brightness caused by motion, the agreement between reconstructed frame gradients and event activity is imposed as in Eq. (16).16\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathcal{L}}_{ef}=\sum\:_{p}{\parallel\:\nabla\:{\widehat{I}}_{t}\left(p\right)-{E}_{t}\left(p\right)\parallel\:}_{1}$$\end{document} where, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\nabla\:{\widehat{I}}_{t}\left(p\right)\:$$\end{document} denotes the spatial gradient magnitude of the reconstructed frame at pixel p; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{E}_{t}\left(p\right)$$\end{document} represents the normalized event activity accumulated at pixel p within the corresponding temporal window; This loss has the benefit of ensuring that the predicted depth and pose produce motions consistent reconstructions that are consistent with brightness variations caused by events, which is especially important with fast-motion and low-light applications of this study. The depth changes between predictions of arbitrary successive depths are imposed to avoid irregular depths. This constraint is given as in Eq. (17).17\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathcal{L}}_{temp}=\sum\:_{p}{\parallel\:{D}_{t}\left(p\right)-{D}_{t+1}\left(p\right)\parallel\:}_{1}$$\end{document} Here, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{D}_{t}\left(p\right)$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{D}_{t+1}\left(p\right)$$\end{document} denote the predicted depth values at pixel p for two consecutive time steps t and t+1, respectively. This is a temporal smoothness term that facilitates stationary scene areas and depth variation in dynamic ones that enhance the accuracy of continuous depth estimation in real-time applications. In order to regularize depth forecasting with geometrical boundaries, an edge-sensitive smoothness constraint is used. It is illustrated as in Eq. (18).18\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathcal{L}}_{smooth}=\sum\:_{p}\left(\mid\:{\partial\:}_{x}{D}_{t}\left(p\right)\mid\:{e}^{-\mid\:{\partial\:}_{x}{I}_{t}\left(p\right)\mid\:}+\mid\:{\partial\:}_{y}{D}_{t}\left(p\right)\mid\:{e}^{-\mid\:{\partial\:}_{y}{I}_{t}\left(p\right)\mid\:}\right)$$\end{document} where, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\partial\:}_{x}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\partial\:}_{y}$$\end{document} denote spatial derivatives along horizontal and vertical image directions, respectively; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mid\:{\partial\:}_{x}{D}_{t}\left(p\right)\mid\:$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mid\:{\partial\:}_{y}{D}_{t}\left(p\right)\mid\:$$\end{document} represent depth gradients; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mid\:{\partial\:}_{x}{I}_{t}\left(p\right)\mid\:$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mid\:{\partial\:}_{y}{I}_{t}\left(p\right)\mid\:$$\end{document} correspond to intensity gradients of the reference frame. The exponential weighting minimizes smoothing at sharp edges of the image and the depth discontinuity at object boundaries are retained. Together, these self-supervised constraints inform the suggested event-frame model in order to acquire geometrically consistent and temporally stable depth representations with a collective exploitation of frame appearance, event-driven motion signals and camera geometry. This model allows the effective depth learning without ground-truth annotations, which is the main goal of the proposed study. Inference for event–frame depth estimation The presented framework is based on the inference of dense depth maps and relative camera motion of completely unseen sequences, using the trained Event Frame Self-Attention Transformer in the absence of any additional supervision. Each input consists of a low-frame-rate intensity image \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{I}_{t}$$\end{document}and its temporally aligned event segment \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathcal{E}}_{t}$$\end{document}, corresponding to the time interval \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:[{T}_{t},{T}_{t+1})$$\end{document}. The preprocessing modules used during training, which include patch extraction, tokenization, event voxelization, and spatial-temporal normalization, are applied to the new input in the same manner in order to maintain the consistency of feature representation. Once the input tokens are generated, they are fed into the EF-SAT layer, which performs modality-specific encoding, cross-frame-rate self-attention, and fusion to produce a unified latent representation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathrm{U}}_{t}$$\end{document}. Such representation is a synthesis of the static views of the intensity frame and dynamic motion cues of the event stream. The attention procedure within the EF-SAT layer enables the model to match clusters of events happening rapidly to their spatial counterparts in the frame, and thus to apply depth inference in areas where the intensity frames only contain limited information. The joint latent representation undergoes the processing of depth and poses decoders. The fused latent representation is then processed by the depth and pose decoders. The depth decoder outputs a dense depth map \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{D}_{t}\in\:{\mathbb{R}}^{H\times\:W}$$\end{document}, where H and W are the spatial dimensions of the input frame. Each pixel \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{D}_{t}\left(p\right)$$\end{document} represents the estimated distance from the camera to the scene point projected onto pixel p. Simultaneously, the pose decoder predicts the relative camera transformation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathrm{T}}_{t\to\:t+1}\in\:SE\left(3\right)$$\end{document}, consisting of a rotation matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathrm{R}}_{t\to\:t+1}$$\end{document}and a translation vector \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathrm{t}}_{t\to\:t+1}$$\end{document}, which describe the rigid motion of the camera between frames t and t+1. These predictions do not only allow depth estimation of the current frame, but they also present vital data in terms of temporal and spatial consistency of the frame with the next frame. Inferential mapping can be mathematically expressed as in Eq. (19).19\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{D}_{t},{\mathrm{T}}_{t\to\:t+1}={f}_{EF\mathrm{-}SAT}\left({I}_{t},{\mathcal{E}}_{t}\right)$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{f}_{EF\mathrm{-}SAT}(\cdot\:)$$\end{document} represents the end-to-end forward pass through the trained Event–Frame Self-Attention Transformer, including the encoder, attention, and decoder stages. Here, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{I}_{t}$$\end{document}is the input intensity frame, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathcal{E}}_{t}$$\end{document} is the corresponding event segment, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{D}_{t}$$\end{document} is the predicted dense depth map, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{\mathrm{T}}_{t\to\:t+1}$$\end{document} is the estimated relative camera pose. The model does not need to have any ground-truth depth or pose in the process of inference, but only learned representations and attention-based relationships between event and frame modalities. This enables the system to extrapolate to new scenes and dynamic motions. The depth maps as predicted can be used directly in 3D reconstruction, obstacle detection, or SLAM applications, whereas relative camera poses allow the strength of accurate temporal alignment and motion estimation in real-time applications. Moreover, event information is integrated to ensure that the system is highly temporally fidelity to fine-scale motion which would otherwise be lost in low-frame-rate image sequences. Algorithm 1Self-supervised event–frame depth estimation using EF-SAT. Self-supervised event–frame depth estimation using EF-SAT. The entire process of self-supervised depth estimation with the proposed EF-SAT is described in algorithm 1. The algorithm initially processes aligned intensity images and event videos to produce patch level encodings that have spatial-temporal positional encoding. These embeddings are cross frame rate self-attended and fused to learn global spatial-temporal dependencies between event and frame modalities. The obtained consolidated representations are inferred to forecast dense depth maps and relative poses of the camera. The self-supervised learning is attained through the enforcement of photometric consistency, event frame alignment, temporal depth consistency and smoothness constraints that allow learning of depth accurately without ground-truth annotations. Fig. 3Flowchart of the proposed system. Flowchart of the proposed system. Figure 3 shows the entire pipeline of the implementation process of the proposed self-supervised depth estimation framework based on EF-SAT. It starts with the synchronized acquisition of event streams and intensity frames and the preprocessing and tokenization to produce spatial temporal representations. Event frame Self-attention transformer then encodes and combines multimodal features using cross-frame rate attention. The latent representation is a unified representation that is decoded into depth and camera pose. Self-supervised constraints are used during the training process with view synthesis and loss computation and iterative parameter updates. When inferencing, the system just gives dense depth maps which allow real-time inference without supervision by ground truth. This study is novel because it integrates event and frame modalities through a principled transformer-based attentional mechanism which is explicitly designed to perform self-supervised depth learning. As opposed to traditional depth prediction frameworks that only use frame-based or convolutional models with narrow receptive field, the presented framework proposes a cross-frame-rate self-attention scheme that explicitly captures the concept of global spatial-temporal interactions between asynchronous event streams and synchronous intensity frames. This design allows event information that is sensitive to motion to inform depth inference where frame-based designs usually fail which occur in low-light conditions or during high-speed camera motion. Moreover, the single representation of transformers does not require any hand-tuned fusion heuristic, which means that the model can learn the modality interactions in an adaptive manner. Unlike the earlier event-based depth learning algorithms which rely on supervised labels or isolated event processing, this paper has realized depth estimation entirely based on self-supervised constraints based on geometric consistency. The prediction of the depth and camera position is co-located in the same latent space, which further reinforces the temporal and the geometric reliability. The proposed methodology can be used to enhance the state of the art in event-frame depth estimation by integrating cross-modal attention, self-supervised learning, and transformer-based global reasoning in a single framework. The combined design distinguishes the study from current methods and forms a scalable basis of the strong depth perception in a real-world dynamic setting. Result and discussion The proposed EF-SAT model has a good and stable performance in self-supervised depth estimation using both streams of events and intensity frames. In various sequences, the model generates rich depth images with intact spatial features and little temporal variations, which assert the usefulness of cross-frame-rate self-attention in the representation of dynamic scene geometry. The qualitative visualizations show the model the capability to preserve sharpness in object contours and time continuity whereas quantitative analysis shows consistent convergence and balanced loss reduction among a variety of self-supervised constraints. Moreover, the framework is practically computationally efficient to enable real-time inference without impairing the quality of prediction. The findings confirm that the proposed self-supervised event-frame fusion method can produce precise, temporally consistent and computationally efficient depth estimates, which is appropriate to be used in real-life dynamic visual perception problems. Table 2Simulation and complexity parameters.ParameterValue/SettingFrame resolution240 × 180Event accumulation window10–20 msEvent temporal bins5Patch size16 × 16Embedding dimension256Transformer layers6Attention heads8Feed-forward dimension1024Batch size4Training epochs40Learning rate1 × 10−4OptimizerAdamPhotometric loss weight1.0Event–frame alignment weight0.5Temporal consistency weight0.3Smoothness weight0.1Total parameters34.1 MFLOPs62.7 GGPU memory usage9.4 GBInference frame rate30 FPSHardware platformNVIDIA RTX 3080 GPU Table 2Simulation and complexity parameters.ParameterValue/SettingFrame resolution240 × 180Event accumulation window10–20 msEvent temporal bins5Patch size16 × 16Embedding dimension256Transformer layers6Attention heads8Feed-forward dimension1024Batch size4Training epochs40Learning rate1 × 10−4OptimizerAdamPhotometric loss weight1.0Event–frame alignment weight0.5Temporal consistency weight0.3Smoothness weight0.1Total parameters34.1 MFLOPs62.7 GGPU memory usage9.4 GBInference frame rate30 FPSHardware platformNVIDIA RTX 3080 GPU Simulation and complexity parameters. The proposed EF-SAT framework summary is listed in Table 2, where the architectural configuration, training settings and computational characteristics are summarized. Besides inference speed, the table also features total parameter count, FLOPs, GPU memory usage and hardware specifications to offer a comprehensive assessment of computational efficiency. The reported complexity was found to be proof of a framework that is real-time but has less computational requirement. Depth prediction quality and temporal stability The qualitative stability of depth predictions of the proposed EF-SAT framework and their temporal stability is evaluated in this section. The model can generate spatially coherent depth images that maintain object boundary and scene structure across successive frames even in scenarios where there is rapid motion of objects and changes in the illumination. The temporal stability analysis shows that there is minimal depth variation in time, which suggests that there is a good investment in event-based motion cues and cross frame rate self-attention. The event stream integration has allowed the model to capture the fine grain dynamics that are not represented in a traditional frame only approach. These findings affirm that the self-supervised approach that is proposed offers reliable depth estimation and preservation of temporal coherence that is required in real-time perception in dynamic situations. Table 3Temporal depth consistency statistics.SequenceMean depth varianceStd. deviationTemporal stability scoreSeq-010.0180.0060.92Seq-020.0210.0070.90Seq-030.0190.0050.93Seq-040.0230.0080.89 Table 3Temporal depth consistency statistics.SequenceMean depth varianceStd. deviationTemporal stability scoreSeq-010.0180.0060.92Seq-020.0210.0070.90Seq-030.0190.0050.93Seq-040.0230.0080.89 Temporal depth consistency statistics. Table 3 shows quantitative statistics of depth predictions in the proposed EF-SAT framework in terms of temporal stability. The metrics are the capability of the model to have consistent depth estimates between the successive frames with minimal temporal variation despite high motion. The low variance and gradual changes imply that the combination of event-driven movement information and cross-frame rate self-attentions are useful in providing the enforcement of temporal coherence. These findings affirm that the suggested self-supervised learning plan maintains depth consistency across time, which is of paramount importance in real-time tasks like navigation, dynamic scene perception and robotic perception. Fig. 4Temporal depth consistency. Temporal depth consistency. Figure 4 demonstrates the consistency in the temporal depth of the proposed EF-SAT model with time. The decrease and stabilization in the depth variance are a positive indication that the model is capable of making temporally consistent depth estimates between successive frames. Such a behaviour proves that the cross-frame-rate self-attention mechanism is an example of the mechanism that helps to put temporal constraints without any special monitoring. Traditional event-driven motion information provides the model with a depth stability even when the scene dynamics are high. This is a highly important feature of real-time use, as in robotic navigation and autonomous perception. The identified stability shows that the suggested self-supervised learning approach is effective to reduce the temporal flickering, which proves the validity of the suggested solution. Fig. 5Depth error under varying camera motion speeds. Depth error under varying camera motion speeds. The correlation between the camera motion speed and depth estimation error is analysed in Fig. 5. The findings show that the error in depth rankings gradually increases with the speed of motion showing that it performs well despite high camera movement. The error growth control brings out the benefit of event-based motion cues, giving information with high-temporal-resolution that could not be obtained in traditional frame-based cues. This is a strength of the EF-SAT attention mechanism that confirms its ability to deal with dynamic scenes with motions blurred and illumination variations eminent. The findings indicate that the suggested framework still has a robust depth estimation in a broad spectrum of motion setups, which supports the appropriateness of the proposed framework in real-world, high-speed perception tasks. Event–frame fusion effectiveness The effectiveness of the proposed event-frame fusion strategy applied using the EF-SAT framework is analysed in this section. The findings prove that the combination of frame-based spatial data and event-based temporal dynamics can greatly contribute to the quality of depth estimation. The self-attention mechanism is able to impose adaptive significance on event and frame properties and can learn to represent them vigorously with demanding motion and lighting environments. There are better experimental findings of structural consistency and less ambiguity in depth prediction than with a single-modality learning. The effective combination of asynchronous streams of events and synchronous streams of frame data proves that the suggested fusion method is effective in capturing complementary information, thus enhancing depth inference in dynamic real-life tasks. Fig. 6Event contribution analysis under different motion regimes. Event contribution analysis under different motion regimes. Figure 6 shows the event data contribution to the accuracy of depth estimation in different regimes of motion. These findings have shown that the suggested EF-SAT structure retains high depth accuracy within the low-motions scenario and shows a lower performance deterioration with the increase of motion intensity. This action brings out the strength of the event-based sensing that has the benefit of capturing fine-grained changes in time that can be easily lost in frame-based methods. Event information in medium and high motion conditions contributes a great amount to robustness through offering motion-resilient cues. These findings affirm the postulated event-frame self-attention process is indeed effective in using event information to enhance reliability of depth estimation especially in dynamic environments. Fig. 7Attention weight distribution of the EF-SAT module. Attention weight distribution of the EF-SAT module. The concentration of the weight of attention learned by the proposed EF-SAT is shown in Fig. 7. The normalized contribution between the event-frame tokens and the heads of attention is the output of each cell and the numerical values are explicitly presented to increase interpretability. The non-uniform, but organized distribution shows that the model is selective in focusing on informative event-based cues of motion and also suppressing redundant features of frames. This adaptive weighting behaviour proves that the EF-SAT mechanism is a good capturing mechanism of the cross-frame-rate dependencies that are essential to self-supervised depth learning. The distinct separation of the attention heads represents the presence of strong feature discrimination, which confirms the hypothesis that the suggested transformer-based fusion approach improves the learning of representations and provides a significant contribution to the overall depth estimation of the research. Computational efficiency and real-time feasibility The efficiency of the computations and the real-time practicability of the suggested EF-SAT framework are assessed. The analysis is done on the inference latency, the module-wise runtime distribution, and the scalability when the input resolution is varied. Findings indicate that the event-frame fusion transformer-based architecture entails low overheads and an efficient implementation. The attention mechanism is an effective way to balance between representational richness and cost in a computation, and allows stable real-time performance. The experimented run-time performance of the given architecture proves that it can be deployed to the time-sensitive system of robotic perception and autonomous navigation. On the whole, the suggested model provides a successful trade-off of accuracy and efficiency to justify its relevance in practical event-frame depth estimation tasks. Fig. 8Inference time versus sequence length. Inference time versus sequence length. Figure 8 demonstrates that the offered EF-SAT framework is scalable by examining the inference time in relation to the sequence length. The findings show that the inference time virtually increases linearly, which means that the transformer-based architecture can efficiently process time. This effect proves that the hypothesized event-frame self-attention system is a more suitable means of controlling long-term dependencies without generating too much computation overhead. The fact that the inference latency increase is controlled shows the appropriateness of the suggested approach to real-time and near-real-time applications. The EF-SAT model supports predictable inference behaviour at different sequence lengths, which guarantees the expected behaviour in dynamic conditions, supporting the viability of the EF-SAT model in use in event-based depth estimation systems. Table 4Computational complexity table.ModelParameters (M) ↓FLOPs (G) ↓FPS ↑SRFNet2642.891.418CAGRU2037.274.622Transformer1558.4126.316EVEN2740.683.520Proposed EF-SAT34.162.730 Table 4Computational complexity table.ModelParameters (M) ↓FLOPs (G) ↓FPS ↑SRFNet2642.891.418CAGRU2037.274.622Transformer1558.4126.316EVEN2740.683.520Proposed EF-SAT34.162.730 Computational complexity table. The computational complexity comparison between the proposed EF-SAT framework and the previously existing CNN/RNN and Transformer based approach is shown in Table 4. The proposed method resulted in reduced number of parameters and FLOPs while ensuring real-time inference speed of 30 frames per second (fps). The light-weight cross-frame-rate self-attention mechanism not only significantly reduced the computational burden but also enhanced the inference efficiency, without affecting the accuracy of depth estimation and temporal motion consistency. Quantitative performance analysis The effectiveness of the suggested EF-SAT framework to learn depth representations in a completely self-supervised environment quantitatively measured. The behaviour analysis aims at training behaviour, its predictability, and behavioural consistency across unseen sequences instead of the reliance on ground-truth supervision. The learning curves that were observed show that convergence is stable, which shows that self-supervised constraints are effective in directing the optimization process. The depth predictions are also stable in time sequences, which is one of the indications of the model of the coherent geometry of the scene. The combination of event driven motion cues in the transformer-based fusion mechanism also adds credibility to the depth inference, especially in dynamic scenarios, which justifies the strength and efficacy of the proposed method. Table 5Contribution of individual self-supervised loss components.Loss componentInitial valueFinal valueReduction (%)Photometric consistency0.4120.09277.7Event–frame alignment0.3650.08177.8Temporal consistency0.2840.06776.4Depth smoothness0.1920.04974.5 Table 5Contribution of individual self-supervised loss components.Loss componentInitial valueFinal valueReduction (%)Photometric consistency0.4120.09277.7Event–frame alignment0.3650.08177.8Temporal consistency0.2840.06776.4Depth smoothness0.1920.04974.5 Contribution of individual self-supervised loss components. Table 5 shows the separate contribution of each component of self-supervised loss to the performance of depth prediction. Photometric reconstruction, event-frame alignment, temporal consistency and smoothness losses are ranked apart to bring out their individual contributions. The evidence shows that each of the components has a complementary function: photometric and event-alignment losses help to control the correct geometry, temporal consistency helps to maintain consistency between the frames, and smoothness regularization helps to maintain edge patterns. These statistics confirm that the loss combination formulation is effective in overseeing the EF-SAT model and facilitates depth estimation reliably in the absence of ground-truth supervision and strengthens the usefulness and effectiveness of the implemented approach. Fig. 9Individual self-supervised loss component trends. Individual self-supervised loss component trends. Figure 9 shows how the components of self-supervised loss of individuals vary during training. Loss terms reduce observably, which means that there is a successful coordination between various supervisory signals. The photometric and event frame alignment losses have a rapid early convergence, which proves that the model is able to learn motion-coherent depth cues using event data. The losses in temporal consistency and smoothness reduce at a lower rate, which guarantees that depth predictions are stable and coherent within time. This balanced action proves that there is no single constraint that prevails the optimization process. The fact that all the loss terms are complementary shows the applicability of the proposed self-supervised formulation in directing EF-SAT model to deep and geometrically consistent estimation. Table 6Ego-motion prediction performance.ModelATERPESRFNet260.1420.089Transformer150.1180.073CAGRU200.1270.081Proposed EF-SAT0.0840.048 Table 6Ego-motion prediction performance.ModelATERPESRFNet260.1420.089Transformer150.1180.073CAGRU200.1270.081Proposed EF-SAT0.0840.048 Ego-motion prediction performance. The absolute trajectory error (ATE) and relative pose error (RPE) are used to assess the capability of the proposed EF-SAT framework for predicting ego-motion, as shown in Table 6. Compared with the existing methods, the proposed method has the minimum trajectory and motion estimation errors. Self-attention across time and across frame rates helps to more consistently predict motion and makes the self-supervised depth–pose learning framework more reliable. Comparative evaluation The proposed EF-SAT framework shows better performance than the base frame-only and event-only techniques. The model achieves superior and more temporally coherent depth maps by cross-frame-rate self-attention between high-temporal-resolution event streams and intensity frames. Quantitative measurements show reduced errors in depth and stronger stability between successive frames, and qualitative visualizations show that there is better preservation of object boundaries and scene structures. Combining those event and frame information enables the model to be used to deal with both rapid motion and low-light situations. In general, these findings confirm that the suggested methodology is better than the traditional ones, which proves its strength and relevance to real-life conditions that are dynamic. Table 7Comparative depth evaluation results.ModelTraining protocolDepth error (AbsRel) ↓RMSE ↓SRFNet26Supervised0.5130.687EVEN27Supervised0.2804.845Transformer15Self-supervised0.1340.274CAGRU20Self-supervised0.2080.298Proposed EF-SATSelf-supervised0.1140.163 Table 7Comparative depth evaluation results.ModelTraining protocolDepth error (AbsRel) ↓RMSE ↓SRFNet26Supervised0.5130.687EVEN27Supervised0.2804.845Transformer15Self-supervised0.1340.274CAGRU20Self-supervised0.2080.298Proposed EF-SATSelf-supervised0.1140.163 Comparative depth evaluation results. Fig. 10Performance comparison across various models. Performance comparison across various models. Table 7 and Fig. 10 shows the comparative results of the proposed EF-SAT framework with the existing supervised and self-supervised depth estimation approaches with consistent evaluation setting on DAVIS-240 C dataset. Training protocol information included in the comparison, explicit to increase transparency and academic rigor. The proposed EF-SAT offered the best AbsRel (0.114) and RMSE (0.163), which showed that this method had greater geometric consistency and depth prediction accuracy. The findings verify both cross-frame-rate self-attention and event-frame fusion to boost the performance for self-supervised depth estimation in dynamic scenes. Table 8Comparative depth evaluation results.DatasetScene typeAbsRelRMSEGeneralization performanceDAVIS-240 CIndoor dynamic0.1140.163HighMVSEC28Indoor/outdoor driving0.1280.192StableDSEC29Outdoor autonomous driving0.1360.214Robust Table 8Comparative depth evaluation results.DatasetScene typeAbsRelRMSEGeneralization performanceDAVIS-240 CIndoor dynamic0.1140.163HighMVSEC28Indoor/outdoor driving0.1280.192StableDSEC29Outdoor autonomous driving0.1360.214Robust Comparative depth evaluation results. Table 8 illustrates how well the proposed EF-SAT framework is able to transfer the concepts to other event-based datasets. The model ensures good depth estimation accuracy even under complex outdoor driving scenes and dynamic motion changes in both MVSEC and DSEC data sets. The AbsRel and RMSE are also slightly higher due to larger scene complexity and illumination variations and high-speed motion, which further demonstrates the robustness of the proposed self-supervised event-frame transformer framework. Ablation study The ablation experiment analyses the effect of important elements in the proposed EF-SAT framework. Avoiding the presence of event information causes a significant loss in depth accuracy, which supports the significance of motion cues with high temporal resolution. The exclusion of cross-frame-rate self-attention leads to impaired temporal stability which validates its contribution in harmonizing the asynchronous representations of events and frames. On the same note, the absence of temporal consistency constraints enhances the change in depth of the frames. The full EF-SAT setup is always outperforming all the reduced versions and proves that the combination of event-frame fusion, transformer-based attention, and self-supervised constraints are necessary to achieve stable and time-synchronous depth estimation. Table 9Ablation study of the proposed EF-SAT framework.MethodDepth error (AbsRel)RMSEFrame-only depth estimation0.1620.289Event-only depth estimation0.1490.261Frame + Event fusion (No self-attention)0.1340.214Frame + Event transformer (No cross-frame-rate modelling)0.1230.191Proposed EF-SAT (Frame + Event, self-supervised)0.1140.163 Table 9Ablation study of the proposed EF-SAT framework.MethodDepth error (AbsRel)RMSEFrame-only depth estimation0.1620.289Event-only depth estimation0.1490.261Frame + Event fusion (No self-attention)0.1340.214Frame + Event transformer (No cross-frame-rate modelling)0.1230.191Proposed EF-SAT (Frame + Event, self-supervised)0.1140.163 Ablation study of the proposed EF-SAT framework. Table 9 presents the ablation study that assesses the contribution of each individual component in the suggested framework EF-SAT. Findings show that event data inclusion is very effective in enhancing depth accuracy and stability of time as compared to frame-only baseline. The cross-frame-rate self-attention introduction further improves performance, which is an effective model to reflect the asynchronous performance between events and frames. Self-supervised constraints have been found to cause performance degradation, and they are important in directing depth learning in the absence of ground truth. The entire EF-SAT setup is the most accurate in-depth error and temporal variation and it definitively confirms that event-driven cues, attention-based fusion and self-supervised learning are essential to the whole efficiency of the suggested technique. Discussion This study shows that the combination of event streams and intensity frames using a self-supervised transformer architecture permits the estimation of depth without ground-truth oversight and makes the estimation of depth accurate and temporally stable. The suggested EF-SAT model is consistent in providing depth predictions in dynamic environments. The success of the EF-SAT can be explained by the fact that it is able to utilize complementary spatial information in frames and high-temporal-resolution motion data in events. This synergy enables the model to deal with high motion, change of illumination and motion blur better than single modality methods. Every element of the framework helps in bringing value to the performance. Motion sensitivity is offered by event-based representations which give up structural context provided by frame-based cues. The self-attention scheme is useful in the feature integration of cross-frames-rate to allow coherent representations of depth. Event streams are discretized during the pre-processing stage but the proposed framework preserves the essential temporal motion patterns using the temporal bin encoding as well as the positional-aware attention learning. While voxelization helps to make the computation more tractable and the optimization of the transformer more stable, some temporal smoothing of extremely high-frequency motions may occur due to quantization over fixed bins. However, such significant spatio-temporal inconsistencies and asynchronies are well maintained for reliable cross-frame-rate depth estimation. Event-only EF-SAT, as compared to frame-only and conventional fusion-based deep learning models, has lower depth errors and higher temporal consistency. Due to transformer-based fusion, global context modelling is possible, which is frequently lacking in CNN-based baselines. The weight distribution of attention shows that the model dynamically lays emphasis on event information in high-motion regions and frame information in static regions that enhances understanding of fusion behaviour. The model is robust to different intensity of motion and this implies that it is stable in response to abrupt changes in a scene which generally causes poor monocular depth estimation. Even though the trained models are trained on one dataset, the modality-agnostic design and self-supervised formulation is promising of generalizing to non-homogeneous environments. In general, EF-SAT provides a good paradigm of self-supervised learning of event-frame depth. However, performance may be affected by extreme event noise, and large-scale cross-dataset validation remains a direction for future work. Conclusion and future work The proposed study introduced a self-supervised event-frame depth learning model, which is based on an EF-SAT. The suggested method is successful in combining both asynchronous streams of events and synchronous intensity frames to teach depth representations without ground-truth supervision. The framework overcomes issues related to motion blur, variation in illumination and sparse temporal sampling when performing depth estimation in traditional frame-based depth estimation, by exploiting the cross-frame-rate self-attention. The outcomes of the experiments prove that the suggested EF-SAT model can be used to obtain consistent predictions of depth in a dynamic scene with a high degree of temporal consistency. The model is efficient in preparing event-based motion cues to boost the level of depth estimates especially where there is rapid motion. Quantitative and qualitative results indicate that the current deep learning-based depth estimates use better performance than the existing deep learning-based depth estimates. The main findings in this research are the development of an integrated self-supervised learning framework that aims to learn event-frame depth, the presentation of a transformer-based cross-frame-rate attention system, and the proof of the practicality to work in real time without affecting the depth quality. The framework suggested can be used in the real-time where robust depth estimation in dynamic environments is essential, including autonomous navigation, robotic perception, and augmented reality. The model used now presupposes that there is correct temporal synchronization between event streams and frames. Also, the performance can be reduced when the event conditions are very noisy. The research will be continued in the future by investigating the extension of the proposed EF-SAT framework to more complex and diverse real-life situations. The combination of dynamic object segmentation may also be seen as one of the promising directions to be taken further in order to enhance depth estimation in highly non-rigid environments. Multi-camera and stereo event-frame systems can also be made to fit into the model and increase the consistency of scales to the geometric accuracy. Also, by investigating light variants of transformers and hardware-conscious optimization techniques, it will become possible to use it on edge and embedded platforms. Another useful direction that has been discovered to enhance reliability of safety-critical applications is to incorporate uncertainty estimation and robustness analysis in the face of extreme noise and illumination conditions.