Robust pose estimation which guarantees positive depths Authors: Chun Li, John E. McInroy Published: 2023 ABSTRACT In the area of 3D computer vision, the ability to estimate pose between two cameras under high noise levels while maintaining small reprojection errors reflects the robustness of such pose estimation algorithms. Moreover, maintaining positive depth constraint is another challenging task. Unfortunately, current pose estimation algorithms are often sensitive to noise/outliers and do not always guarantee positive depths. As a standalone task, these algorithms perform a positive sign check and simply discard the points with negative depths after the algorithms are executed. These algorithms do not integrate positive depth constraints into the algorithms themselves. Instead, they do it afterwards. Here, from a comprehensive mathematical derivation, we propose a novel pose estimation algorithm that integrates positive depth constraint into the algorithm itself by estimating the depths directly. The algorithm was competitive in producing small reprojection errors when compared to the state-of-the-art algorithms under both synthetic and real-world tests, while most importantly guaranteeing positive depths. Introduction Pose estimation is widely and thoroughly studied in the field of computer vision, which tackles the problem of solving relative pose between cameras or world coordinate systems. It is proven useful in various real-world computer vision application scenarios like Structure from Motion (SfM)1, Simultaneous Localization and Mapping (SLAM)2, Light Detection and Ranging (LIDAR)3,4, autonomous navigation5,6, Augmented Reality (AR)7,8, and so on. From a model-wise perspective, the input of pose estimation problem is camera observations, usually denoted as uncalibrated pixel coordinates or calibrated image coordinates. The output is pose parameters between coordinate systems, specifically, rotation matrix, translation vector, depth, etc. Most importantly, the integral part that connects the input and the output is known as pose estimation algorithm. In literature, pose estimation is often interchangeably used with terms such as Perspective-n-Point (PnP). Therefore, PnP algorithms are also considered pose estimation algorithms, which comprise both 3D-2D and 2D-2D PnP algorithms according to different nature of observations. Over the decades, pose estimation algorithms or PnP algorithms have evolved dramatically, on both algorithmic design and performance. According to the nature of the observations, state-of-the-art (SOTA) algorithms can be categorized into 2D-2D and 3D-2D methods, which require intrinsic camera information for pose estimation. Representative 2D-2D algorithms include Eight-point (8-pt)9, normalized Eight-point (8-pt (norm))10, Minimum-eigenvalue (Min-eig)11, Five-point (5-pt)12,13 algorithm. These 2D-2D algorithms accept 2D-2D calibrated image coordinates as inputs. On the other hand, popular 3D-2D algorithms include EPnP14, EPnP-GN15, DLS16, RPnP17, LHM18, DLT19, OPnP20, MLPnP21,22, and (R)EPPnP23 algorithms, and also more recently, the SQPnP24, CPnP25, Uncertain-PnP or PnP(L)26, and QPEPs27. These solvers usually try to achieve certain optimization goals. For example, due to the linearity nature, 8-pt and 8-pt (norm) focus on execution efficiency, which is simple and fast; 5-pt algorithm only requires five correspondence pairs to recover pose, extending its compatibility in more scenarios; EPnP algorithm demonstrates competitive performance with low computational cost; RPnP algorithm further improves the performance over EPnP with similar computational cost, excelling in translation estimates; OPnP performs well for all variables including rotation, translation, and depth, even under presence of noise; (R)EPPnP incorporates outlier rejection scheme into the algorithm itself, circumventing the time-consuming Random Sample Consensus (RANSAC)28–32 procedure, which is frequently utilized to remove outliers from the input dataset; and finally, MLPnP is a maximum likelihood solution to the PnP problem that includes image observation uncertainties to improve accuracy. Such uncertainties do occur in real-world scenarios since not all points are measured with equal certainty. For example, imperfections on sensors could lead to such uncertainties, resulting shifts on pixels. To an extent, MLPnP can also handle outliers, with outlier scrutinization process also embedded inside the algorithm. The more recent SQPnP treats PnP problem as a non-linear quadratic program (NLQP), and always determines the global minima of the PnP problem “for any number of input correspondences”, with tolerance on coplanar configurations24. The CPnP algorithm is another recent progress that introduces consistent estimator for the camera pose via a bias-eliminated closed-form solution, which later is refined by the constrained Gauss–Newton (GN) iterations25. The techniques used to integrate these two algorithms into our work for comparison can be found in Methods. Despite the distinctive merits of the above algorithms, these SOTA algorithms encounter common yet demanding challenges, mainly on the need of noise tolerance and positive depth guarantee (positive depth constraint). On the one hand, being able to tolerate high noise levels shows the algorithms’ robustness and potential of real-world adoption; on the other hand, since physically depth signs must all be positive (points are in the front of camera), being capable of guaranteeing positive depth estimates eliminates the need of cheirality check33 (sign check) after algorithm execution, which is a redundant, time-consuming yet unavoidable task that requires multiple steps of manual intervention. Such intervention consists of removing extraneous solutions containing negative depth estimates, or even requires elimination of certain data corresponding to negative depths, which causes data loss. Unfortunately, current SOTA algorithms can hardly maintain robustness, while also conforming to positive depth constraint. For instance, the highly robust OPnP algorithm has no guarantee that the depth estimates will be all positive. Therefore, being able to be robust to noise while simultaneously guaranteeing positive depth is a critical concern for current pose estimation algorithms. Here, through complete mathematical derivation, a pose estimation algorithm called Pos-dep (or Min-eig-Depths) is proposed. The algorithm is specifically designed to guarantee positive depth estimates while demonstrating robustness to noise changes, yielding small reprojection errors compared to the SOTA. The cheirality check process that examines the signs of the depths is integrated into the proposed algorithm itself, circumventing extra checks on the signs. Based on a solid mathematical backbone, the algorithm iteratively solves the rotation, and the eigenvector of a data matrix that contains all positive entries as the depth vector. The proposed algorithm obtains positive depth estimates even before the pose is estimated, making cheirality check and solution selection no longer necessary by providing a unique solution. In addition, the algorithm is robust and adapts to noisy conditions in terms of increasing noise intensities and presence of outliers, showing its robustness to noise changes. Results We will first present the results obtained using simulated data. Then, we present the results obtained in real-world scenarios. We separated the experiment on synthetic data (simulation) into three subsections: varying the noise levels and varying the percentage of outliers to fully test on the algorithms' robustness to noise and outliers, and varying the number of points to test the performance and computation time when more points are added. Varying the noise levels We performed the experiments on the range of standard deviation (std) of zero-mean Gaussian noise covering from 1e − 7 to 1e − 1 in calibrated image coordinates. To cover such wide noise spectrum, we used “semilog” plots for the x-axis. The y-axis was the percentage of good estimates so that the higher the plot, the better the performance. The number of iterations done at each noise level was 100. The number of points was ten. Figures 1, 2, 3 and 4 show all algorithms’ percentages of good rotation, translation, reprojection, and depth estimates under varying noise levels. By saying “good”, we are referring that the estimation errors are less than some set thresholds. It can be seen from Fig. 1 that the Pos-dep was among the best ones, it only downgraded after the noise std was beyond 1e − 2. The DLT and (R)EPPnP were the worst. We can also see that our Pos-dep algorithm was the best among all 2D-2D algorithms, keeping at a percentage of 100 when the noise was below 1e − 2. After that, it also had the slowest degrading rate compared to other 2D-2D algorithms. The SQPnP outperformed CPnP when the noise std exceeds 1e − 2.Figure 1Rotation (< 30◦) vs. noise levels.Figure 2Translation (< 30%) vs. noise levels.Figure 3Reprojection (< 0.5) vs. noise levels.Figure 4Depth (< 30%) vs. noise levels. Rotation (< 30◦) vs. noise levels. Translation (< 30%) vs. noise levels. Reprojection (< 0.5) vs. noise levels. Depth (< 30%) vs. noise levels. For translations, we see from Fig. 2 that the Pos-dep stayed at 100% line until 3e − 3, after that it kept comparable with 3D-2D algorithms such as MLPnP and EPnP. Among all 2D-2D algorithms, our Pos-dep beat all others when the noise was below 3e − 3; when the noise was beyond this level, our algorithm still remained comparable to the Min-eig, while beating others, including the 8-pt (norm). The SQPnP outperformed CPnP when the noise std exceeds 1e − 2. From Fig. 3, we see for reprojections, of all the algorithms the Pos-dep performed the best (stayed up at the 100% line), including the 3D-2D PnP algorithms and the recent SQPnP and CPnP. In fact, being capable of reaching and maintaining small reprojection errors across the noise spectrum is one of the main advantages of the Pos-dep algorithm. The SQPnP outperformed CPnP when the noise std exceeds 2e − 2 (0.02). For depth estimates, we see from Fig. 4 that overall, the 3D-2D algorithms were better than the 2D-2D, with DLT and (R)EPPnP being the exceptions. Among all 2D-2D algorithms, our Pos-dep again beat all other 2D-2D algorithms, especially at low-medium noise levels. The SQPnP outperformed CPnP when the noise std exceeds 1e − 2. In terms of negative depth instances, we can clearly see from Fig. 5 that out of 100 tests on each noise level, the Pos-dep stayed all the way flat at the zero-instance line, guaranteeing positive depth estimates. The SQPnP algorithm overall has fewer instances of negative depths than the CPnP algorithm.Figure 5Negative depth instances vs. noise levels. Negative depth instances vs. noise levels. The computation time of Pos-dep was consistently around 6 ms across the noise spectrum, lower than the OPnP algorithm which was around 10 ms (Fig. 6). The runtime was recorded using MATLAB on a PC with 16 GB RAM and 6-core Intel CPU with 2.6 GHz clock frequency.Figure 6Computation time vs. noise levels. Computation time vs. noise levels. Limited by the length of this paper, for more comprehensive results that involve testing with smaller thresholds and comparison within the 3D-2D and 2D-2D algorithmic pools only, please refer to pages 194–210 of this thesis34. In addition to the percentage of good estimates, mean and median estimation errors are also the metrics actively used in this area, as is performed in the OPnP paper20. Here, at each noise level, 100 runs again are independently executed. The mean and median error at each noise level is calculated. The DLS algorithm is excluded in the plot since it is out of bounds. Close-up plots are shown next to the originals. These results are shown in Fig. 7, which includes the results of mean and median rotation, translation, reprojection, and depth errors.Figure 7Mean and median estimation errors when varying the noise levels. (a) Mean rotation error, a close-up is shown to the right; (b) Median rotation error, a close-up is shown to the right; (c) Mean translation error, a close-up is shown to the right; (d) Median translation error, a close-up is shown to the right; (e) Mean reprojection error, a close-up is shown to the right; (f) Median reprojection error, a close-up is shown to the right; (g) Mean depth error, a close-up is shown to the right; (h) Median depth error, a close-up is shown to the right. Mean and median estimation errors when varying the noise levels. (a) Mean rotation error, a close-up is shown to the right; (b) Median rotation error, a close-up is shown to the right; (c) Mean translation error, a close-up is shown to the right; (d) Median translation error, a close-up is shown to the right; (e) Mean reprojection error, a close-up is shown to the right; (f) Median reprojection error, a close-up is shown to the right; (g) Mean depth error, a close-up is shown to the right; (h) Median depth error, a close-up is shown to the right. For rotations, in Fig. 7a and b, when the noise std is at 1e − 2, the median rotation error of Pos-dep was around 5 degrees. For translations, in Fig. 7c and d, when the noise std is at 1e − 2, the median translation error of Pos-dep was around 10%. For reprojections, we can see from Fig. 7e and f that both the mean and median reprojection errors of Pos-dep were among the smallest, compared with other algorithms. For depths, we can see from Fig. 7g and h that when the noise std is below 1e − 2, the depth errors were below 30%, which was used as the threshold to differentiate “good” estimates. From Fig. 7, we also found the OPnP and the SQPnP had similar performance, and overall performed the best in the algorithm pool. Varying the percentage of outliers We define the outliers to be the data points corrupted by noise with a std of 0.1 (1e-1). We chose the percentages to be from 10 to 100% with a 10% increment. The total number of points (inliers and outliers combined) was 20. Again, 100 tests were independently performed at each outlier rate. While a proportion of the points were assigned as outliers, other points were not corrupted by noise and were still considered “clean data”. We followed the similar evaluation method used in varying the noise levels. We can see from Fig. 8 that the Pos-dep was comparable to the EPnP-GN and better than EPnP. The OPnP and the SQPnP stayed at 100% and overlapped with each other, both were better than the CPnP algorithm. At high outlier rates such as 70%, the Pos-dep still achieved more than 80 good rotation estimates, out of 100.Figure 8Rotation (< 30◦) vs. outlier rate. Rotation (< 30◦) vs. outlier rate. Figure 9 shows that our Pos-dep again was the best among all 2D-2D algorithms. The LHM, OPnP, and SQPnP had similar performance. The SQPnP outperformed CPnP. Overall, the Pos-dep was comparable to EPnP-GN. At 60% outlier rate, the Pos-dep outperformed CPnP algorithm. At high outlier rates such as 90% and 100%, the Pos-dep can also compete with the CPnP algorithm.Figure 9Translation (< 30%) vs. outlier rate. Translation (< 30%) vs. outlier rate. Figure 10 shows that for reprojections, our Pos-dep performed exceptionally well among the algorithms. It stayed at 100%. OPnP, LHM, RPnP, and SQPnP were the next. The SQPnP outperformed CPnP. The threshold used to classify good reprojection was 0.5, measured in calibrated image coordinates.Figure 10Reprojection (< 0.5) vs. outlier rate. Reprojection (< 0.5) vs. outlier rate. Figure 11 shows that for depth errors the Pos-dep again beat all 2D-2D algorithms. The LHM, OPnP, and SQPnP were the best among all algorithms. The SQPnP outperformed the CPnP here. The threshold was 30%. Our Pos-dep was the only 2D-2D algorithm that could be comparable with the 3D-2D algorithms. In fact, it beat MLPnP when the outlier rate was beyond 30%. From Fig. 11, it can be seen our algorithm almost formed a “separating line” between the 3D-2D and 2D-2D algorithms, following the EPnP algorithm.Figure 11Depth (< 30%) vs. outlier rate. Depth (< 30%) vs. outlier rate. For negative depth instances, we see from Fig. 12 that the Pos-dep stayed at the bottom of the graph all the time, meaning it recorded zero instances of negative depths. We can also see that interestingly, the negative depth instances gradually increased as the outlier rate increased. At 100% outlier rate, roughly at least 30 instances of negative depths were reported for the SOTA algorithms. The SQPnP algorithm overall has fewer instances of negative depths than the CPnP algorithm.Figure 12Negative depth instances. Negative depth instances. In terms of computation time, see Fig. 13. In theory, the computation time of Pos-dep should be nearly constant. Although there exists a slight increase of the computation time of Pos-dep, since the number of points were fixed at 20 and the computation cost of Pos-dep only depends on point numbers, we argue such increase was within the margin of reasonable system errors. For instance, as experiments going on, the machine might have gone through possible thermal throttling with reduced clock frequency that impacted the CPU’s performance. The computation time of Pos-dep ranges approximately from 20 to 40 ms, the difference is 20 ms. The OPnP stayed nearly consistently at 10 ms.Figure 13Computation time vs. outlier rate. Computation time vs. outlier rate. Similar to varying the noise levels, we also evaluated the performance by studying the mean and median estimation errors including the rotation, translation, reprojection, and depths. Again, at each outlier rate (percentage), 100 runs are independently executed. The mean and median errors at each outlier rate are calculated. The configurations such as the noise std, the choices of percentages, and the number of total points are the same as those presented in Figs. 8, 9, 10, 11, 12 and 13. The DLS algorithm is excluded since it is out of bounds. The results are shown in Fig. 14.Figure 14Mean and median estimation errors when varying the percentage of outliers. (a) Mean and median rotation errors; (b) Mean and median translation errors; (c) Mean and median reprojection errors; (d) Mean and median depth errors. Mean and median estimation errors when varying the percentage of outliers. (a) Mean and median rotation errors; (b) Mean and median translation errors; (c) Mean and median reprojection errors; (d) Mean and median depth errors. From Fig. 14a, we see that for rotations, the mean error of Pos-dep was comparable to that of EPnP-GN, while the median error was among the best that stayed below 20% for all outlier rates. For translations, in Fig. 14b, when the outlier rate is below 40%, the mean translation error is below 30%, and the median translation error is below 20%. In extreme circumstances such as 90% outlier rate, the mean rotation, median rotation, mean translation, and median translation errors of Pos-dep were around 20 degrees, 20 degrees, 40%, and 30%, respectively. For reprojections, from Fig. 14c we see the Pos-dep outperformed other algorithms across the outlier rates in both mean and median errors. At 100% outlier rate, the median error of Pos-dep was around 0.1, smaller than all other algorithms. For depth estimates, from Fig. 14d we see the mean errors of Pos-dep were among the best, and the median errors were the best among all 2D-2D algorithms and beat (R)EPPnP. When the outlier rate exceeds 50%, the median depth errors of Pos-dep were comparable to the most recent CPnP algorithm, although admittedly, the recent algorithms SQPnP and CPnP were among the best ones in each subplot of Fig. 14. In this experiment, the noise std used was 0.1 in calibrated coordinates, which could comfortably transform the raw data into outliers. Varying the number of points The 8-pt algorithm requires the minimum number of points used is 8. Hence, we chose the number of points to be from 8 to 30, with an increment of 2. Figures 15, 16, 17 and 18 show the errors of rotation, translation, reprojection, and depth, respectively.Figure 15Rotation errors vs. number of points.Figure 16Translation errors vs. number of points.Figure 17Reprojection errors vs. number of points.Figure 18Depth errors vs. number of points. Rotation errors vs. number of points. Translation errors vs. number of points. Reprojection errors vs. number of points. Depth errors vs. number of points. We can see from Fig. 15 that when the number of points exceeds 16, the proposed algorithm performed well (the higher the better), and the CPnP performed even better; the SQPnP only reported around 20% of good estimates. For translations we can see from Fig. 16 that the Pos-dep was only behind RPnP, which stayed above 90% in good estimates for all point numbers. The SQPnP barely reached 50%. For reprojection, we see from Fig. 17 that the Pos-dep again stayed high and was only behind EPnP-GN, which maintained above 90% in good estimates when point number was greater than 10. For depth estimates, from Fig. 18 we see the Pos-dep was better than the SQPnP algorithm, except only at point numbers 24, 28, and 30. The proposed algorithm stayed around 90% in good estimates across all point numbers. The proposed algorithm reported zero instance of negative depths all the time (Fig. 19), while on average (R)EPPnP and DLT reported more than 20 negative instances out of a total of 100 when point number exceeds 18, more than other algorithms.Figure 19Negative depth instances vs. number of points. Negative depth instances vs. number of points. Figure 20 shows the computation time (runtime) of the algorithms versus the number of points. We see that the runtime of our algorithm almost “linearly” increased. This is expected due to the mathematical nature of our algorithm’s \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{D}}}_{{\varvec{R}}}$$\end{document}DR matrix, please see Eq. (23) in Methods for definition. The \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{D}}}_{{\varvec{R}}}$$\end{document}DR matrix is of dimension \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$2n\times 2n$$\end{document}2n×2n, where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n$$\end{document}n is the number of points. As the number of points increase, the dimension of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{D}}}_{{\varvec{R}}}$$\end{document}DR would also increase linearly, resulting in an increase of computation time when calculating eigenvectors and eigenvalues of the matrix. It is noted that even when the number of points is 30, the proposed algorithm still reported a runtime below 40 ms. For each point number, 100 runs were independently performed at each point number and the averaged runtime over these runs was reported for each point number. The reader is also encouraged to refer to pages 212 to 215 of the thesis34 for more comprehensive results when varying the number of correspondence points.Figure 20Computation time vs. number of points. Computation time vs. number of points. In real applications, hundreds of or even thousands of points may exist. Therefore, we also studied the variations of computation time when the number of points is ranging from 100 to 1000, with a 100 increment. The runtime was recorded using MATLAB on the same PC with 16 GB RAM and 6-core Intel CPU with 2.6 GHz clock frequency. We compared our proposed algorithm with the more performant OPnP algorithm in Fig. 21.Figure 21Runtime comparison between Pos-dep and OPnP when larger numbers of points are used. (a) Runtime comparison when the number of points ranges from 100 to 1000; (b) A close-up of the OPnP algorithm; (c) A close-up of the Pos-dep when the number of points equaling 100, 200, and 300; (d) A close-up of the Pos-dep when the number of points equaling 400, 500, and 600; (e) A close-up of the Pos-dep when the number of points equaling 700, 800, 900, and 1000. Runtime comparison between Pos-dep and OPnP when larger numbers of points are used. (a) Runtime comparison when the number of points ranges from 100 to 1000; (b) A close-up of the OPnP algorithm; (c) A close-up of the Pos-dep when the number of points equaling 100, 200, and 300; (d) A close-up of the Pos-dep when the number of points equaling 400, 500, and 600; (e) A close-up of the Pos-dep when the number of points equaling 700, 800, 900, and 1000. We observe from Fig. 21 that similar to the case shown in Fig. 20, the runtime of Pos-dep increases with the increasing number of points. As discussed before, an increase of point numbers \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n$$\end{document}n would cause an increase on the dimension of the matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{D}}}_{{\varvec{R}}}$$\end{document}DR, which is of size \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$2n\times 2n$$\end{document}2n×2n. Thus, the eigenvalue and eigenvector computation of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{D}}}_{{\varvec{R}}}$$\end{document}DR would be more computationally expensive. We also see the OPnP presents a slight runtime increase, ranging from approximately 0.01 s to over 0.03 s. We speculate such increase (only about 20 ms difference) was within the margin of system error caused by the heating-up machine with thermal throttling happening under prolonged computation process. Again, in Fig. 21 the averaged runtime over 100 independent runs at each point number was reported. We noticed the thresholds chosen may be too small (such as 1e-14) for certain scenarios in the above experiments. Therefore, just like the first two scenarios, we also studied the mean and median estimation errors, which is thoroughly used in the OPnP work20. Also, in the above experiments, we did not add noise to the data when varying the number of points. While in the OPnP work20, zero-mean Gaussian noise with standard deviation of 2 pixels was added. Hence, we also added zero-mean Gaussian noise with a fixed standard deviation of 0.01 in calibrated image coordinates when studying the mean and median errors, which corresponds to noise with a standard deviation of 8 pixels. For each data point, 100 independent runs were performed, and the mean/median errors were reported. In20, 4 to 15 points with an increment of 1 are used; whereas in our experiments, to meet the requirement of each algorithm and to be consistent with the previous results, 8 to 30 points with an increment of 1 were used. The experimental results are shown in Fig. 22. The 5-pt and DLS were excluded from the graphs since they were out of bounds comparing to other algorithms.Figure 22Mean and median rotation, translation, reprojection, and depth errors when varying the number of points under fixed noise of 8 pixels. (a) Mean rotation error, a close-up is shown to the right; (b) Median rotation error, a close-up is shown to the right; (c) Mean translation error, a close-up is shown to the right; (d) Median translation error, a close-up is shown to the right; (e) Mean reprojection error, a close-up is shown to the right; (f) Median reprojection error, a close-up is shown to the right; (g) Mean depth error, a close-up is shown to the right; (h) Median depth error, a close-up is shown to the right. Mean and median rotation, translation, reprojection, and depth errors when varying the number of points under fixed noise of 8 pixels. (a) Mean rotation error, a close-up is shown to the right; (b) Median rotation error, a close-up is shown to the right; (c) Mean translation error, a close-up is shown to the right; (d) Median translation error, a close-up is shown to the right; (e) Mean reprojection error, a close-up is shown to the right; (f) Median reprojection error, a close-up is shown to the right; (g) Mean depth error, a close-up is shown to the right; (h) Median depth error, a close-up is shown to the right. For rotation errors, in Fig. 22a, when the point number reaches 13, the Pos-dep starts to yield a mean rotation error less than 5 degrees, which is considered the threshold of a good estimate in this study. In Fig. 22b, the median rotation error starts to fall below 5 degrees when the point number reaches 11. For translations, in Fig. 22c, when the point number reaches 13, the Pos-dep starts to yield a mean translation error less than 10%, which is considered the threshold of a good estimate in this study. In Fig. 22d, the median translation error starts to fall below 10% when the point number reaches 10. For reprojection errors, it can be seen from Fig. 22e and f that the Pos-dep was among the best, and when the point number reaches 14 and beyond, the Pos-dep reported an averaged reprojection error of less than 0.02 at each number of points. For depth errors, we see from Fig. 22g that the Pos-dep was better than (R)EPPnP; from Fig. 22h we see that when point number reaches 15, the Pos starts to yield median depth errors less 10%, which is considered the threshold of “good” estimate in this study. Through Fig. 22, we see the recent algorithms, the SQPnP and CPnP, both performed well, staying low among the algorithms across all point numbers. On real-world scenarios The experiments on real-world data contain four scenarios/cases. The first and the second utilize standard datasets available online with ground truth provided35. The third case was conducted on a rigid box image taken from a mirrorless camera (SONY α6000). The fourth case was conducted on a satellite mockup image taken from an iPhone rear camera (iPhone SE2). Both cameras were self-calibrated using MATLAB’s camera calibration toolbox and a checkerboard pattern to obtain the intrinsic parameters. Results on the standard datasets For the first scenario, we used ten pairs of successive dinosaur images from the dataset. We ran the testing for each image pair in the dataset and calculated the mean and median estimation errors. We removed the points not on the object and removed duplicate points, making the total number of correspondence points to be consistent (around ten) for each pair of images (we increased the number of points in the RANSAC experiment). These correspondences were generated by SIFT36 algorithm. We summarized the results in tabular forms and marked our algorithm in green. The images used from the Dino dataset can be found in Fig. S1 of the Supplementary Information. We see from Table 1 that our algorithm reported an average of 0.5917 degrees, beating all algorithms except DLS, RPnP and OPnP; the median rotation error (0.6038) was below the 5-degree threshold for good estimates, being slightly better than the CPnP. The CPnP performed better than SQPnP.Table 1Rotation errors in degrees (Dino).Image pairs1&22&33&44&55&66&77&88&99&1010&11MeanMedian8-pt7.88848.03637.61817.78937.92527.82427.90519.99237.88125.72777.85887.88488-pt (norm)8.23286.49167.22637.86957.78167.76077.68325.94467.87665.02367.18917.7219Min-eig3.68517.40067.40383.598618.25858.17879.113510.290235.834918.159512.19238.6461Pos-dep0.43470.54060.68980.27620.32440.22020.76440.66700.93411.06590.59170.60385-pt7.66487.623475.45267.74307.68787.321210.12156.503668.18476.483320.47867.6763EPnP1.36312.05970.78810.74177.30723.53891.91251.86330.96405.49322.60321.8879EPnP-GN1.91447.62340.78810.10150.52850.38991.91250.09600.96402.32020.93330.8760DLS0.13580.23170.07350.07770.25870.05730.43450.03610.09710.29840.17010.1164RPnP0.07820.26460.59150.15320.30320.05990.41070.11180.10210.22680.23020.1900LHM13.881915.575021.455122.044518.745019.699917.043914.61929.02740.615315.270716.3094DLT179.2787179.4312179.89100.240534.5961179.58321.9966179.7438179.41266.8974112.1071179.3456OPnP0.13570.23150.07340.07720.25900.05680.43300.03630.09620.30120.17000.1159MLPnP0.13450.22600.07450.0765103.48020.05580.42990.0369103.480361.493726.94880.1803(R)EPPnP0.34721.69421.14621.2608178.52840.62327.6186178.201036.958917.871042.42504.6564SQPnP1.62671.62531.17903.489911.28442.93923.28431.64227.363030.08666.45213.1117CPnP0.14820.34780.07081.002622.80590.20810.96580.04551.86079.07283.65280.6568Significant values are in bold. Table 1Rotation errors in degrees (Dino).Image pairs1&22&33&44&55&66&77&88&99&1010&11MeanMedian8-pt7.88848.03637.61817.78937.92527.82427.90519.99237.88125.72777.85887.88488-pt (norm)8.23286.49167.22637.86957.78167.76077.68325.94467.87665.02367.18917.7219Min-eig3.68517.40067.40383.598618.25858.17879.113510.290235.834918.159512.19238.6461Pos-dep0.43470.54060.68980.27620.32440.22020.76440.66700.93411.06590.59170.60385-pt7.66487.623475.45267.74307.68787.321210.12156.503668.18476.483320.47867.6763EPnP1.36312.05970.78810.74177.30723.53891.91251.86330.96405.49322.60321.8879EPnP-GN1.91447.62340.78810.10150.52850.38991.91250.09600.96402.32020.93330.8760DLS0.13580.23170.07350.07770.25870.05730.43450.03610.09710.29840.17010.1164RPnP0.07820.26460.59150.15320.30320.05990.41070.11180.10210.22680.23020.1900LHM13.881915.575021.455122.044518.745019.699917.043914.61929.02740.615315.270716.3094DLT179.2787179.4312179.89100.240534.5961179.58321.9966179.7438179.41266.8974112.1071179.3456OPnP0.13570.23150.07340.07720.25900.05680.43300.03630.09620.30120.17000.1159MLPnP0.13450.22600.07450.0765103.48020.05580.42990.0369103.480361.493726.94880.1803(R)EPPnP0.34721.69421.14621.2608178.52840.62327.6186178.201036.958917.871042.42504.6564SQPnP1.62671.62531.17903.489911.28442.93923.28431.64227.363030.08666.45213.1117CPnP0.14820.34780.07081.002622.80590.20810.96580.04551.86079.07283.65280.6568Significant values are in bold. Rotation errors in degrees (Dino). Significant values are in bold. For translations, from Table 2 we see our Pos-dep reported 5.2030% for translation error, outperforming all except the DLS, RPnP and OPnP. The median error (4.4400%) was below the 10% threshold for good estimates. This time, the SQPnP performed better than the CPnP in both mean and median errors.Table 2Translation errors in percentage (Dino).Image pairs1&22&33&44&55&66&77&88&99&1010&11MeanMedian8-pt147.9800143.9900137.7300140.8600148.9800147.7700139.9700129.9900137.3400122.6800139.7290140.41508-pt (norm)159.2400129.6500134.4900142.7500149.6700147.8900134.02006.3200137.2900118.5800125.9900135.8900Min-eig13.5500139.0200115.64002.040049.4500196.710035.7400199.890096.080047.250089.537072.7650Pos-dep4.28003.11004.60002.30004.60001.19003.96008.50006.080013.41005.20304.44005-pt161.9300133.6100159.3300137.7600138.9000115.400041.090019.090074.370069.0600105.0540124.5050EPnP12.180024.56009.91005.120060.820032.980025.23009.540011.690037.750022.978018.3700EPnP-GN3.550023.54009.91001.28007.26004.960025.23001.070011.690030.660011.91508.5850DLS1.75001.83000.46000.85002.05000.64004.08000.42000.74003.98001.68001.3000RPnP0.78003.87006.48002.01002.31000.71004.58001.52000.66003.07002.59902.1600LHM31.860084.650092.3000105.6400133.2500131.4300146.8700100.3700138.23007.790097.2390103.0050DLT174.4600166.0000161.67002.470073.0300138.870026.0700129.610097.790040.9300101.0900113.7000OPnP1.75001.82000.46000.85002.04000.63004.06000.42000.73004.01001.67701.3000MLPnP1.73001.74000.46000.8400132.34000.60004.05000.4400143.9700189.120047.52901.7350(R)EPPnP4.830020.990013.200020.6000154.390012.620066.4400165.910092.740084.650063.637043.7150SQPnP17.970021.470013.660046.280049.710028.710040.29005.8700114.5800183.580052.212034.5000CPnP2.83005.39001.530094.1500148.530019.520069.36002.020092.6100105.570054.151044.4400Significant values are in bold. Table 2Translation errors in percentage (Dino).Image pairs1&22&33&44&55&66&77&88&99&1010&11MeanMedian8-pt147.9800143.9900137.7300140.8600148.9800147.7700139.9700129.9900137.3400122.6800139.7290140.41508-pt (norm)159.2400129.6500134.4900142.7500149.6700147.8900134.02006.3200137.2900118.5800125.9900135.8900Min-eig13.5500139.0200115.64002.040049.4500196.710035.7400199.890096.080047.250089.537072.7650Pos-dep4.28003.11004.60002.30004.60001.19003.96008.50006.080013.41005.20304.44005-pt161.9300133.6100159.3300137.7600138.9000115.400041.090019.090074.370069.0600105.0540124.5050EPnP12.180024.56009.91005.120060.820032.980025.23009.540011.690037.750022.978018.3700EPnP-GN3.550023.54009.91001.28007.26004.960025.23001.070011.690030.660011.91508.5850DLS1.75001.83000.46000.85002.05000.64004.08000.42000.74003.98001.68001.3000RPnP0.78003.87006.48002.01002.31000.71004.58001.52000.66003.07002.59902.1600LHM31.860084.650092.3000105.6400133.2500131.4300146.8700100.3700138.23007.790097.2390103.0050DLT174.4600166.0000161.67002.470073.0300138.870026.0700129.610097.790040.9300101.0900113.7000OPnP1.75001.82000.46000.85002.04000.63004.06000.42000.73004.01001.67701.3000MLPnP1.73001.74000.46000.8400132.34000.60004.05000.4400143.9700189.120047.52901.7350(R)EPPnP4.830020.990013.200020.6000154.390012.620066.4400165.910092.740084.650063.637043.7150SQPnP17.970021.470013.660046.280049.710028.710040.29005.8700114.5800183.580052.212034.5000CPnP2.83005.39001.530094.1500148.530019.520069.36002.020092.6100105.570054.151044.4400Significant values are in bold. Translation errors in percentage (Dino). Significant values are in bold. We can see from Table 3 that the reprojection error of our Pos-dep was 0.0001, which is the smallest among all algorithms and matches the OPnP. The median error of Pos-dep (0.0001) was also small compared to other algorithms. The SQPnP had smaller reprojection errors in both mean and median than the CPnP.Table 3Reprojection errors in calibrated units (Dino).Image pairs1&22&33&44&55&66&77&88&99&1010&11MeanMedian8-pt0.00350.00710.00320.03020.00240.00240.00120.05510.00390.01160.01210.00378-pt (norm)0.00660.00810.00620.00550.00750.00200.02530.00700.00380.00870.00810.0068Min-eig00.00010.0001000000.00020.00010.00010Pos-dep00.00010.00030.000100.00010.000100.00010.00040.00010.00015-pt0.00010.00050.00090.00100.00030.0001000.00040.00030.00040.0003EPnP0.00020.00020.00010.00010.00040.00020.00010.00010.00020.00160.00030.0002EPnP-GN00.00030.0001000.00010.000100.00020.00070.00020.0001DLS00.000100000.000100.00010.00030.00010RPnP00.00010.00010000.000100.00010.00030.00010.0001LHM0.00080.00120.00140.00120.00070.00110.00100.00070.00190.00030.00100.0011DLT0.00030.00040.000300.00180.00050.00010.00070.00210.00150.00080.0005OPnP00.000100000.000100.00010.00030.00010MLPnP00.0001000.186800.000102.59880.26160.30470.0001(R)EPPnP0.00020.00080.00010.00400.01820.00420.00130.04600.01420.00620.00950.0041SQPnP0.00040.00020.00010.00030.00070.00010.00030.00010.00160.00220.00060.0003CPnP0.00070.00120.00010.04850.43280.00420.03420.00040.05280.23780.08130.0192Significant values are in bold. Table 3Reprojection errors in calibrated units (Dino).Image pairs1&22&33&44&55&66&77&88&99&1010&11MeanMedian8-pt0.00350.00710.00320.03020.00240.00240.00120.05510.00390.01160.01210.00378-pt (norm)0.00660.00810.00620.00550.00750.00200.02530.00700.00380.00870.00810.0068Min-eig00.00010.0001000000.00020.00010.00010Pos-dep00.00010.00030.000100.00010.000100.00010.00040.00010.00015-pt0.00010.00050.00090.00100.00030.0001000.00040.00030.00040.0003EPnP0.00020.00020.00010.00010.00040.00020.00010.00010.00020.00160.00030.0002EPnP-GN00.00030.0001000.00010.000100.00020.00070.00020.0001DLS00.000100000.000100.00010.00030.00010RPnP00.00010.00010000.000100.00010.00030.00010.0001LHM0.00080.00120.00140.00120.00070.00110.00100.00070.00190.00030.00100.0011DLT0.00030.00040.000300.00180.00050.00010.00070.00210.00150.00080.0005OPnP00.000100000.000100.00010.00030.00010MLPnP00.0001000.186800.000102.59880.26160.30470.0001(R)EPPnP0.00020.00080.00010.00400.01820.00420.00130.04600.01420.00620.00950.0041SQPnP0.00040.00020.00010.00030.00070.00010.00030.00010.00160.00220.00060.0003CPnP0.00070.00120.00010.04850.43280.00420.03420.00040.05280.23780.08130.0192Significant values are in bold. Reprojection errors in calibrated units (Dino). Significant values are in bold. For depth errors, we rounded the results to integers. We can see from Table 4 that both the mean and median errors of our Pos-dep were 4%, only larger than the DLS, RPnP and OPnP algorithms, and were comparable to the EPnP-GN. The SQPnP had smaller mean depth error than the CPnP, while its median error was larger than CPnP.Table 4Depth errors in percentage (Dino).Image pairs1&22&33&44&55&66&77&88&99&1010&11MeanMedian8-pt66133621687143862098036401891078-pt (norm)12816352222132221163303742136122Min-eig9134016259069136552181806939691Pos-dep37120194104445-pt5217125689401638397954609893951309502EPnP16162837556287382116EPnP-GN3820126171143DLS021030301011RPnP114030201011LHM64616969586150563125260DLT10010010028010071001004473100OPnP021030301011MLPnP021093030107108312(R)EPPnP466310044110183694224SQPnP91655846132740682720CPnP121461091261331323514Significant values are in bold. Table 4Depth errors in percentage (Dino).Image pairs1&22&33&44&55&66&77&88&99&1010&11MeanMedian8-pt66133621687143862098036401891078-pt (norm)12816352222132221163303742136122Min-eig9134016259069136552181806939691Pos-dep37120194104445-pt5217125689401638397954609893951309502EPnP16162837556287382116EPnP-GN3820126171143DLS021030301011RPnP114030201011LHM64616969586150563125260DLT10010010028010071001004473100OPnP021030301011MLPnP021093030107108312(R)EPPnP466310044110183694224SQPnP91655846132740682720CPnP121461091261331323514Significant values are in bold. Depth errors in percentage (Dino). Significant values are in bold. In terms of negative depth instances, by checking the experimental data we see that out of ten pairs of images, negative depths occurred in two image pairs for the (R)EPPnP, three image pairs for the 8-pt (norm), four image pairs for the 5-pt, six image pairs for the DLT and the 8-pt algorithms. All other algorithms, including our Pos-dep, recorded zero negative instances. For the Dino dataset, the computation time of each algorithm on each image pair, is summarized in Table 5. We see the Pos-dep yielded 0.0066 s (6.6 ms) for both the mean and median computation time on single image pair, which is comparable to the SQPnP algorithm, which yielded 6.5 ms for the mean time, and 6.8 ms for the median time. The computation time of SQPnP was nearly half of that of CPnP.Table 5Computation time on each image pair (Dino).Image pairs1&22&33&44&55&66&77&88&99&1010&11MeanMedian8-pt0.01280.01700.01280.01320.01820.01220.01300.01300.01290.01310.01380.01308-pt (norm)0.03530.04190.03440.03410.03710.03320.03410.03450.03420.03620.03550.0345Min-eig0.14650.13520.09630.11970.09740.09390.09500.08990.11220.10970.10960.1036Pos-dep0.00570.00820.00520.00670.00830.00780.00640.00770.00530.00440.00660.00665-pt0.06070.07150.05170.05700.06620.06150.06400.05090.04800.04810.05800.0588EPnP0.02160.03020.02150.02140.02780.02200.02120.02140.02180.02100.02300.0215EPnP-GN0.03420.04170.03340.03310.04070.03460.03290.03430.03530.03460.03550.0344DLS0.05580.05940.05770.05710.06170.05520.05430.05290.05600.05170.05620.0559RPnP0.03160.03470.03110.03830.03560.03380.03250.03210.03450.03420.03380.0340LHM0.02820.02590.02570.02670.02910.02780.02540.02280.02530.02120.02580.0258DLT0.01070.00840.00780.01190.00830.00730.00760.00850.00850.00680.00860.0083OPnP0.08970.07380.06760.08740.07430.05420.07750.05180.08320.06980.07290.0741MLPnP0.10000.10060.09800.10530.09890.10320.10320.10210.09750.08530.09940.1003(R)EPPnP0.01720.02090.02020.01660.01970.02210.01730.01900.01830.01610.01870.0186SQPnP0.00640.00730.00700.00630.00580.00690.00680.00690.00680.00520.00650.0068CPnP0.01300.01340.01470.01230.01440.01320.01350.01290.01420.01160.01330.0133Significant values are in bold. Table 5Computation time on each image pair (Dino).Image pairs1&22&33&44&55&66&77&88&99&1010&11MeanMedian8-pt0.01280.01700.01280.01320.01820.01220.01300.01300.01290.01310.01380.01308-pt (norm)0.03530.04190.03440.03410.03710.03320.03410.03450.03420.03620.03550.0345Min-eig0.14650.13520.09630.11970.09740.09390.09500.08990.11220.10970.10960.1036Pos-dep0.00570.00820.00520.00670.00830.00780.00640.00770.00530.00440.00660.00665-pt0.06070.07150.05170.05700.06620.06150.06400.05090.04800.04810.05800.0588EPnP0.02160.03020.02150.02140.02780.02200.02120.02140.02180.02100.02300.0215EPnP-GN0.03420.04170.03340.03310.04070.03460.03290.03430.03530.03460.03550.0344DLS0.05580.05940.05770.05710.06170.05520.05430.05290.05600.05170.05620.0559RPnP0.03160.03470.03110.03830.03560.03380.03250.03210.03450.03420.03380.0340LHM0.02820.02590.02570.02670.02910.02780.02540.02280.02530.02120.02580.0258DLT0.01070.00840.00780.01190.00830.00730.00760.00850.00850.00680.00860.0083OPnP0.08970.07380.06760.08740.07430.05420.07750.05180.08320.06980.07290.0741MLPnP0.10000.10060.09800.10530.09890.10320.10320.10210.09750.08530.09940.1003(R)EPPnP0.01720.02090.02020.01660.01970.02210.01730.01900.01830.01610.01870.0186SQPnP0.00640.00730.00700.00630.00580.00690.00680.00690.00680.00520.00650.0068CPnP0.01300.01340.01470.01230.01440.01320.01350.01290.01420.01160.01330.0133Significant values are in bold. Computation time on each image pair (Dino). Significant values are in bold. For the second scenario, we tested the Temple dataset from the same website35. Please also refer to pages 227–234 of this thesis34. Recent algorithms such as SQPnP and CPnP were again included. Again, successive image pairs were used. We ran the testing for each image pair in the dataset and calculated the mean and median estimation errors. We removed the points not on the object and removed duplicate points, making the total number of correspondence points to be consistent (around ten) for each pair of images (we increased the number of points in the RANSAC experiment). These correspondences were generated by SIFT36 algorithm. The results are summarized in tabular forms. The images used from the Temple dataset can be found in Fig. S2 of the Supplementary Information. We see from Table 6 that our algorithm reported an average of 1.3111 degrees, beating all algorithms except DLS, EPnP-GN and OPnP; the median rotation error (1.0991) was below the 5-degree threshold for good estimates, and was better than SQPnP. CPnP performed better than SQPnP in both mean and median rotation errors.Table 6Rotation errors in degrees (Temple).Image pairs1&22&33&44&55&66&77&88&99&1010&11MeanMedian8-pt4.85082.04327.30107.598712.46946.39137.71358.18779.82066.46537.28427.44988-pt (norm)4.52751.32650.99271.389112.40495.38724.02238.19415.02136.24674.95124.7744Min-eig0.91159.98772.47669.14840.31008.244012.931160.18222.405119.681612.62788.6962Pos-dep2.45090.92821.03060.89910.79521.16771.37301.51602.41360.53721.31111.09915-pt1.278051.63078.78668.02473.38468.22508.11383.60240.89187.649410.15877.8370EPnP0.54611.12461.25640.18073.24811.57111.45945.64053.71144.03922.27771.5152EPnP-GN0.02470.07420.10210.18070.46590.34851.45941.92520.80351.03020.64140.4072DLS0.02680.05570.32430.25260.25800.44451.35591.18261.02550.40370.53300.3640RPnP0.04480.08321.42020.13560.50480.39861.92101.22982.11480.44300.82960.4739LHM1.53260.74131.27270.68683.89275.64041.90882.48282.62733.79912.45852.1958DLT0.5465179.8900179.7759179.9453179.9914179.8598179.5160179.77823.73076.2678126.9302179.7771OPnP0.02610.05560.32740.25330.25760.44371.35461.18281.02340.39340.53180.3604MLPnP0.02140.04770.34460.25500.25440.43481.3239139.80081.00640.405614.38950.3751(R)EPPnP179.9103179.8130176.97151.7407179.7147178.8965178.8356178.1579176.1635177.7765160.7980178.4968SQPnP0.02600.05550.96620.95512.44922.00081.74719.18804.59810.40412.23901.3567CPnP0.01960.05550.33490.26890.22510.43611.388110.48740.91827.23682.13710.3855Significant values are in bold. Table 6Rotation errors in degrees (Temple).Image pairs1&22&33&44&55&66&77&88&99&1010&11MeanMedian8-pt4.85082.04327.30107.598712.46946.39137.71358.18779.82066.46537.28427.44988-pt (norm)4.52751.32650.99271.389112.40495.38724.02238.19415.02136.24674.95124.7744Min-eig0.91159.98772.47669.14840.31008.244012.931160.18222.405119.681612.62788.6962Pos-dep2.45090.92821.03060.89910.79521.16771.37301.51602.41360.53721.31111.09915-pt1.278051.63078.78668.02473.38468.22508.11383.60240.89187.649410.15877.8370EPnP0.54611.12461.25640.18073.24811.57111.45945.64053.71144.03922.27771.5152EPnP-GN0.02470.07420.10210.18070.46590.34851.45941.92520.80351.03020.64140.4072DLS0.02680.05570.32430.25260.25800.44451.35591.18261.02550.40370.53300.3640RPnP0.04480.08321.42020.13560.50480.39861.92101.22982.11480.44300.82960.4739LHM1.53260.74131.27270.68683.89275.64041.90882.48282.62733.79912.45852.1958DLT0.5465179.8900179.7759179.9453179.9914179.8598179.5160179.77823.73076.2678126.9302179.7771OPnP0.02610.05560.32740.25330.25760.44371.35461.18281.02340.39340.53180.3604MLPnP0.02140.04770.34460.25500.25440.43481.3239139.80081.00640.405614.38950.3751(R)EPPnP179.9103179.8130176.97151.7407179.7147178.8965178.8356178.1579176.1635177.7765160.7980178.4968SQPnP0.02600.05550.96620.95512.44922.00081.74719.18804.59810.40412.23901.3567CPnP0.01960.05550.33490.26890.22510.43611.388110.48740.91827.23682.13710.3855Significant values are in bold. Rotation errors in degrees (Temple). Significant values are in bold. For translations, from Table 7 we see our Pos-dep reported 8.9336% for translation error, outperforming all except the DLS, EPNP-GN and OPnP. The median error (9.9931%) was below the 10% threshold. The SQPnP performed better than the CPnP in mean error, while worse than CPnP in median error.Table 7Translation errors in percentage (Temple).Image pairs1&22&33&44&55&66&77&88&99&1010&11MeanMedian8-pt92.142279.9670138.5780143.4303168.650651.6684146.1642143.298488.6514122.0048117.4545130.29148-pt (norm)92.344468.823833.095611.8773168.375240.867535.3483143.253289.3272120.961780.427479.0755Min-eig1.3116199.93523.1606199.47683.3325194.845747.156177.179617.616767.247081.126257.2015Pos-dep1.27130.466511.19985.26818.786512.386616.018616.462512.20925.26738.93369.99315-pt4.5532194.935248.592097.714439.8143109.4196156.994916.49851.920671.166374.160959.8792EPnP1.94765.948113.93602.180324.23483.668418.088563.824240.684948.596422.310916.0123EPnP-GN0.35171.22421.74132.18035.82584.388918.088523.064610.109113.13728.01125.1074DLS0.36400.68613.85042.88013.21505.547316.969214.910912.62145.19266.62374.5215RPnP0.59361.174017.86510.29456.19544.962523.973615.979425.32465.578310.19415.8868LHM20.17499.737216.12847.785541.706761.319822.937030.694931.208144.856628.654926.8160DLT1.951990.0492162.5359172.1880149.979337.5055123.5760114.665540.808225.674791.8934102.3573OPnP0.35670.68873.89232.88533.21025.536716.951914.924812.59965.05396.61004.4731MLPnP0.30820.57654.13692.90423.17035.424016.562739.651412.38605.21809.03384.6775(R)EPPnP137.8186107.810592.219420.2228111.644486.258987.617080.5449144.1819152.8809102.1199100.0149SQPnP0.36000.680011.22001.670017.92002.680013.5100156.270022.30005.200023.18108.2100CPnP0.37002.95007.75005.22002.88005.720021.4500142.160034.7400121.750034.49906.7350Significant values are in bold. Table 7Translation errors in percentage (Temple).Image pairs1&22&33&44&55&66&77&88&99&1010&11MeanMedian8-pt92.142279.9670138.5780143.4303168.650651.6684146.1642143.298488.6514122.0048117.4545130.29148-pt (norm)92.344468.823833.095611.8773168.375240.867535.3483143.253289.3272120.961780.427479.0755Min-eig1.3116199.93523.1606199.47683.3325194.845747.156177.179617.616767.247081.126257.2015Pos-dep1.27130.466511.19985.26818.786512.386616.018616.462512.20925.26738.93369.99315-pt4.5532194.935248.592097.714439.8143109.4196156.994916.49851.920671.166374.160959.8792EPnP1.94765.948113.93602.180324.23483.668418.088563.824240.684948.596422.310916.0123EPnP-GN0.35171.22421.74132.18035.82584.388918.088523.064610.109113.13728.01125.1074DLS0.36400.68613.85042.88013.21505.547316.969214.910912.62145.19266.62374.5215RPnP0.59361.174017.86510.29456.19544.962523.973615.979425.32465.578310.19415.8868LHM20.17499.737216.12847.785541.706761.319822.937030.694931.208144.856628.654926.8160DLT1.951990.0492162.5359172.1880149.979337.5055123.5760114.665540.808225.674791.8934102.3573OPnP0.35670.68873.89232.88533.21025.536716.951914.924812.59965.05396.61004.4731MLPnP0.30820.57654.13692.90423.17035.424016.562739.651412.38605.21809.03384.6775(R)EPPnP137.8186107.810592.219420.2228111.644486.258987.617080.5449144.1819152.8809102.1199100.0149SQPnP0.36000.680011.22001.670017.92002.680013.5100156.270022.30005.200023.18108.2100CPnP0.37002.95007.75005.22002.88005.720021.4500142.160034.7400121.750034.49906.7350Significant values are in bold. Translation errors in percentage (Temple). Significant values are in bold. We can see from Table 8 that the reprojection error of our Pos-dep was 0.0005, which was among the smallest ones. It was comparable to SQPnP and better than CPnP. The median error of Pos-dep (0.0003) was also among the best, and better than the CPnP. The SQPnP had smaller reprojection errors in both mean and median errors than the CPnP.Table 8Reprojection errors in calibrated units (Temple).Image pairs1&22&33&44&55&66&77&88&99&1010&11MeanMedian8-pt0.00870.00980.01840.03970.00900.02250.01300.00080.05770.01400.01940.01358-pt (norm)0.01650.00690.00640.00080.02470.00280.02290.00080.02780.01360.01230.0102Min-eig00.00040.00010.000300.00010.00030.00080.00020.00060.00030.0003Pos-dep0.00170.00020.00020.00020.00010.00020.00030.00080.00040.00040.00050.00035-pt0.00010.00210.00060.00030.00020.00040.00040.00050.00020.00060.00050.0004EPnP0.00020.00080.00030.00010.00010.00010.00030.00110.00040.00090.00040.0003EPnP-GN0.00010.00010.00030.00010.00010.00010.00030.00080.00050.00090.00030.0002DLS0.00010.00010.00010.000100.00010.00020.00060.00030.00040.00020.0001RPnP0.00010.00010.00020.000100.00010.00020.00090.00030.00040.00020.0002LHM0.00080.00040.00030.00010.00030.00090.00040.00080.00040.00100.00050.0004DLT0.00020.01250.00320.00090.00100.00370.00270.00420.00040.00090.00300.0019OPnP0.00010.00010.00010.000100.00010.00020.00060.00030.00040.00020.0001MLPnP0.00010.00010.00010.000100.00010.00020.25390.00030.00050.02550.0001(R)EPPnP0.06080.00360.11710.00050.13030.05380.08700.04890.09430.03980.06360.0573SQPnP0.00010.00010.00020.00010.00010.00010.00020.00230.00040.00040.00040.0002CPnP0.00010.00030.00090.00050.00010.00010.00170.04520.00480.12650.01800.0007Significant values are in bold. Table 8Reprojection errors in calibrated units (Temple).Image pairs1&22&33&44&55&66&77&88&99&1010&11MeanMedian8-pt0.00870.00980.01840.03970.00900.02250.01300.00080.05770.01400.01940.01358-pt (norm)0.01650.00690.00640.00080.02470.00280.02290.00080.02780.01360.01230.0102Min-eig00.00040.00010.000300.00010.00030.00080.00020.00060.00030.0003Pos-dep0.00170.00020.00020.00020.00010.00020.00030.00080.00040.00040.00050.00035-pt0.00010.00210.00060.00030.00020.00040.00040.00050.00020.00060.00050.0004EPnP0.00020.00080.00030.00010.00010.00010.00030.00110.00040.00090.00040.0003EPnP-GN0.00010.00010.00030.00010.00010.00010.00030.00080.00050.00090.00030.0002DLS0.00010.00010.00010.000100.00010.00020.00060.00030.00040.00020.0001RPnP0.00010.00010.00020.000100.00010.00020.00090.00030.00040.00020.0002LHM0.00080.00040.00030.00010.00030.00090.00040.00080.00040.00100.00050.0004DLT0.00020.01250.00320.00090.00100.00370.00270.00420.00040.00090.00300.0019OPnP0.00010.00010.00010.000100.00010.00020.00060.00030.00040.00020.0001MLPnP0.00010.00010.00010.000100.00010.00020.25390.00030.00050.02550.0001(R)EPPnP0.06080.00360.11710.00050.13030.05380.08700.04890.09430.03980.06360.0573SQPnP0.00010.00010.00020.00010.00010.00010.00020.00230.00040.00040.00040.0002CPnP0.00010.00030.00090.00050.00010.00010.00170.04520.00480.12650.01800.0007Significant values are in bold. Reprojection errors in calibrated units (Temple). Significant values are in bold. For depth errors, we rounded the results to integers. We can see from Table 9 that both the mean and median errors of our Pos-dep were 4%. The mean error of Pos-dep beat those of both the SQPnP and CPnP. The median error of Pos-dep beat that of SQPnP. The CPnP performed better than the SQPnP in terms of both mean and median depth errors.Table 9Depth errors in percentage (Temple).Image pairs1&22&33&44&55&66&77&88&99&1010&11MeanMedian8-pt20123035131708612676215171338-pt (norm)1411411918391020591613413Min-eig1329245614121,02159873368222364Pos-dep41011536742445-pt13824720,27677674313973121937332678EPnP713101552451413881513EPnP-GN011110542321DLS001100111011RPnP001210024111LHM31231518526964DLT7101100100100117100101134178100OPnP001100111011MLPnP0011001881191(R)EPPnP1021001006101100100981029991100SQPnP001113633221593403017CPnP0111001132355191Significant values are in bold. Table 9Depth errors in percentage (Temple).Image pairs1&22&33&44&55&66&77&88&99&1010&11MeanMedian8-pt20123035131708612676215171338-pt (norm)1411411918391020591613413Min-eig1329245614121,02159873368222364Pos-dep41011536742445-pt13824720,27677674313973121937332678EPnP713101552451413881513EPnP-GN011110542321DLS001100111011RPnP001210024111LHM31231518526964DLT7101100100100117100101134178100OPnP001100111011MLPnP0011001881191(R)EPPnP1021001006101100100981029991100SQPnP001113633221593403017CPnP0111001132355191Significant values are in bold. Depth errors in percentage (Temple). Significant values are in bold. In terms of negative depth instances, by checking the experimental data we see that out of ten pairs of images, negative depths occurred in one image pair for the 8-pt (norm) and CPnP, three image pairs for the 8-pt, four image pairs for the 5-pt, eight image pairs for the DLT, nine image pairs for the (R)EPPnP. All other algorithms, including our Pos-dep, recorded zero negative instances. For the Temple dataset, the computation time of each algorithm on each image pair, is summarized in Table 10. We see the Pos-dep yielded 0.0040 s (4 ms) for the mean time, and 0.0036 s (3.6 ms) for the median time. The Pos-dep took less computation time than both the more recent SQPnP and CPnP algorithms. Again, we found the computation time of SQPnP was nearly half of that of CPnP.Table 10Computation time on each image pair (Temple).Image pairs1&22&33&44&55&66&77&88&99&1010&11MeanMedian8-pt0.01430.01480.01340.01460.01400.01660.01780.01390.01410.01440.01480.01438-pt (norm)0.05730.03610.03420.03580.04500.04130.04470.03710.03390.03610.04020.0366Min-eig0.15820.08980.11760.08850.12400.10170.09520.09720.09060.09360.10560.0962Pos-dep0.00170.00300.00210.00630.00270.00290.00740.00570.00430.00410.00400.00365-pt0.04910.06250.05260.05090.04670.04320.04380.05100.06680.04520.05120.0500EPnP0.02050.02570.02010.01970.01990.02000.02330.01980.02360.02690.02200.0203EPnP-GN0.03390.03280.03230.03210.03260.03390.03410.03160.04280.04290.03490.0334DLS0.05000.05020.05040.05080.05390.05030.05060.05100.06230.05500.05250.0507RPnP0.02720.02950.02420.02360.03140.02790.03100.02700.03000.02410.02760.0275LHM0.02160.02490.02010.02130.02400.02090.02360.02110.02330.02250.02230.0221DLT0.00720.00750.00690.00670.00870.00710.00710.00710.00900.00730.00750.0072OPnP0.05240.04830.05380.04990.06280.04660.06580.04670.07230.04640.05450.0512MLPnP0.07630.08180.07270.07290.07730.07450.07480.07830.08200.07780.07680.0768(R)EPPnP0.01660.01860.01640.01600.01800.01660.01760.01310.01730.01560.01660.0166SQPnP0.00540.00640.00520.00570.00630.00550.00740.00530.00590.00520.00580.0056CPnP0.01260.01550.01250.01210.01360.01210.01210.01220.01360.01230.01290.0124Significant values are in bold. Table 10Computation time on each image pair (Temple).Image pairs1&22&33&44&55&66&77&88&99&1010&11MeanMedian8-pt0.01430.01480.01340.01460.01400.01660.01780.01390.01410.01440.01480.01438-pt (norm)0.05730.03610.03420.03580.04500.04130.04470.03710.03390.03610.04020.0366Min-eig0.15820.08980.11760.08850.12400.10170.09520.09720.09060.09360.10560.0962Pos-dep0.00170.00300.00210.00630.00270.00290.00740.00570.00430.00410.00400.00365-pt0.04910.06250.05260.05090.04670.04320.04380.05100.06680.04520.05120.0500EPnP0.02050.02570.02010.01970.01990.02000.02330.01980.02360.02690.02200.0203EPnP-GN0.03390.03280.03230.03210.03260.03390.03410.03160.04280.04290.03490.0334DLS0.05000.05020.05040.05080.05390.05030.05060.05100.06230.05500.05250.0507RPnP0.02720.02950.02420.02360.03140.02790.03100.02700.03000.02410.02760.0275LHM0.02160.02490.02010.02130.02400.02090.02360.02110.02330.02250.02230.0221DLT0.00720.00750.00690.00670.00870.00710.00710.00710.00900.00730.00750.0072OPnP0.05240.04830.05380.04990.06280.04660.06580.04670.07230.04640.05450.0512MLPnP0.07630.08180.07270.07290.07730.07450.07480.07830.08200.07780.07680.0768(R)EPPnP0.01660.01860.01640.01600.01800.01660.01760.01310.01730.01560.01660.0166SQPnP0.00540.00640.00520.00570.00630.00550.00740.00530.00590.00520.00580.0056CPnP0.01260.01550.01250.01210.01360.01210.01210.01220.01360.01230.01290.0124Significant values are in bold. Computation time on each image pair (Temple). Significant values are in bold. Results on the rigid box image For the third scenario, we performed the experiment on a rigid box. To get the 3D-2D matches, we used a SONY α6000 mirrorless camera with a fixed focal length of 30 mm to complete the calibration process using a checkerboard pattern. Then, we used the same calibrated camera without refocusing to capture the image of the rigid box. The original dimension of the image was 6000 (H) by 4000 (V) pixels. We chose ten correspondence points on the surface of the rigid box with MATLAB’s “getpts” control point selection functionality for 2D point selection. Since the ground truth was not known, we transformed the two 3D-2D problem into one 3D-3D problem with ground truth known. The rigid box image with control points (correspondences) selected, as well as the transformation method from 3D-2D to 3D-3D, are presented in Figs. S3 and S4 of Supplementary Information. For rotation errors, we see from Fig. 23 that our Pos-dep was 3.87 degrees, comparable to the 8pt, normalized 8-pt, and MLPnP. The SQPnP and OPnP were at zero errors, better than the CPnP (0.19 degrees).Figure 23Rotation errors in degrees (rigid box). Rotation errors in degrees (rigid box). We see from Fig. 24 that the Pos-dep had good translation errors of 6.48%, lower than the 10% threshold and better than the normalized 8-pt, 5-pt, and MLPnP; the LHM, OPnP, and SQPnP had zero errors, better than the CPnP (0.79%).Figure 24Translation errors in percentage (rigid box). Translation errors in percentage (rigid box). We can see from Fig. 25 that the reprojection error of our Pos-dep was 0.0275, among the smallest and comparable to EPnP’s. The SQPnP and CPnP had similar reprojection errors (0.0116, 0.0125) with the SQPnP performed slightly better.Figure 25Reprojection errors in calibrated units (rigid box). Reprojection errors in calibrated units (rigid box). From Fig. 26, we see the depth error of Pos-dep was 1.16%, much lower than the 10% threshold for good estimates. The SQPnP (0.02%) performed better than the CPnP (0.13%) algorithm in terms of depth errors.Figure 26Depth errors in percentage (rigid box). Depth errors in percentage (rigid box). In the rigid box case, no negative depths were reported for all algorithms. In terms of computation time, the 8-pt was 0.0078 s, the 8-pt norm 0.0282 s, the Min-eig 0.0417 s, the Pos-dep 0.0100 s, the 5-pt 0.0147 s, the EPnP 0.1177 s, the EPnP-GN 0.0155 s, the DLS 0.0322 s, the RPnP 0.0050 s, the LHM 0.0047 s, the DLT 0.0016 s, the OPnP 0.0340 s, the MLPnP 0.0131 s, the (R)EPPnP 0.0083 s, the SQPnP 0.0026 s, and the CPnP was 0.0034 s. The 10 ms-computation time of Pos-dep was most close to those of the MLPnP (13.1 ms) and (R)EPPnP (8.3 ms). The SQPnP (2.6 ms) had less computation time than the CPnP (3.4 ms). Using the estimated results, it is tempting to calculate the 3D reconstructed model of the rigid box and compare it with the original one. This result is included in Fig. S5 of Supplementary Information. Results on the satellite mockup image For the fourth scenario, we performed the experiment on a satellite mockup image. The process to obtain the 3D-2D matches was similar to that in the previous rigid box case. The original dimension of the image this time was 4032 (H) by 3024 (V) pixels. The focal length was fixed at 3.99 mm on the iPhone’s rear-facing camera throughout the calibration process. Since the ground truth was again not known, we applied the same 3D-2D to 3D-3D transform as that used in the rigid box case. The configuration of the satellite mockup with coordinate system setup and selected correspondence points, was shown in Fig. S6 of Supplementary Information. For rotation errors, we see from Fig. 27 that our Pos-dep reported 12.75 degrees. The SQPnP, LHM, and OPnP were at zero errors, better than the CPnP (0.02 degrees).Figure 27Rotation errors in degrees (satellite mockup). Rotation errors in degrees (satellite mockup). We see from Fig. 28 that the Pos-dep reported a translation error of 9.45%, lower than the 10% threshold; the LHM, OPnP, and SQPnP had zero errors, better than the CPnP (1.26%).Figure 28Translation errors in percentage (satellite mockup). Translation errors in percentage (satellite mockup). We can see from Fig. 29 that the reprojection error of our Pos-dep was 0.0240, among the smallest ones and comparable to EPnP. The SQPnP and CPnP had similar reprojection errors (0.0162, 0.0166) with the SQPnP performed slightly better.Figure 29Reprojection errors in calibrated units (satellite mockup). Reprojection errors in calibrated units (satellite mockup). From Fig. 30, we see the depth error of Pos-dep was 2.42%, much lower than the 10% threshold for good estimates. The CPnP (0%) was better than the SQPnP (0.05%) in terms of depth errors.Figure 30Depth errors in percentage (satellite mockup). Depth errors in percentage (satellite mockup). In the satellite mockup case, all algorithms except the 5-pt reported positive depths. In terms of computation time, the 8-pt was 0.0006 s, the 8-pt norm 0.0078 s, the Min-eig 0.0498 s, the Pos-dep 0.0103 s, the 5-pt 0.0037 s, the EPnP 0.0103 s, the EPnP-GN 0.0102 s, the DLS 0.0029 s, the RPnP 0.0025 s, the LHM 0.0034 s, the DLT 0.0005 s, the OPnP 0.0199 s, the MLPnP 0.0055 s, the (R)EPPnP 0.0024 s, the SQPnP 0.0023 s, and the CPnP was 0.0010 s. The 10.3 ms-computation time of Pos-dep was most close to those of the EPnP (10.3 ms) and EPnP-GN (10.2 ms). The CPnP (1 ms) had less computation time than the SQPnP (2.3 ms). Comparing with the RANSAC algorithm In the practice of pose estimation using PnP algorithms, outlier detection and rejection strategies such as RANSAC is often applied and only inliers are used for pose estimation, similar to the (R)EPPnP23 algorithm. To understand how our proposed algorithm performs comparing to the RANSAC algorithm, we self-coded the RANSAC algorithm to remove the outliers and used only inliers to estimate pose. To achieve a tradeoff between accuracy and efficiency, our RANSAC algorithm takes the Min-eig algorithm to iteratively select the consensus set, then it utilizes our proposed algorithm to estimate the pose using only the inliers. All correspondence points, including both the inliers and outliers, are obtained from the 25th and 26th image of the standard dinosaur dataset35 using the SIFT algorithm. Figure 31 shows all the correspondences found with straight lines connecting them. 334 correspondences were obtained in total, which more reflects real-world scenarios where hundreds of points are usually encountered.Figure 31All correspondences found including inliers and outliers. All correspondences found including inliers and outliers. Since the ground truth pose can be directly computed from the dataset, the rotation errors, translation errors, reprojection errors, and depth errors with and without RANSAC can be compared. Without RANSAC, the rotation error was 12.5368 degrees; the translation error was 13.0664%; the depth error from the first image was 5.0166%, and 5.0268% from the second image; the reprojection error was 0.0032 in calibrated image coordinates; the runtime was 4.1672 s (334 points). With RANSAC, 16 correspondence points were randomly selected for the consensus set, and 100 iterations were used to finalize such set. Therefore, the resultant consensus dataset comprising all inliers has 16 data points. The rotation error was 5.1490 degrees; the translation error was 2.5094%; the depth error from the first image was 4.3327%, and 4.4798% from the second image; the reprojection error was 0.00013 in calibrated image coordinates; the runtime was reduced to 0.0126 s due to the reduced number of points. The runtime of the RANSAC process only was 7.5512 s. Illustrative results are shown in Fig. 32, where the averaged depth error of both images, and the cumulative runtime (RANSAC and algorithm execution combined) are shown. For visualization purposes, the reprojection errors are not shown in Fig. 32 and are only numerically compared.Figure 32Accuracy and runtime comparison without and with RANSAC applied. With RANSAC applied before algorithm execution, the accuracy improves at the cost of increased cumulative runtime. Accuracy and runtime comparison without and with RANSAC applied. With RANSAC applied before algorithm execution, the accuracy improves at the cost of increased cumulative runtime. Using RANSAC before algorithm execution, the accuracy improves at the cost of increased cumulative runtime. The accuracy improvement is more obvious in Rotation and Translation Errors than that in Averaged Depth Errors. However, the tradeoff is the increased computation time, with most of the cost consumed in the RANSAC process. Discussion In this work, we presented a positive depth-guaranteed pose estimation algorithm that estimates the camera pose with better or comparable performance to the state-of-the-art algorithms in certain aspects. The proposed Pos-dep algorithm guarantees positive depths. It solves depths with all positive entries directly within the algorithm execution process, instead of later calculating them, which ensures the positivity of depths even before the pose is estimated. With the existence of noise and outliers, our algorithm showed its robustness to various interfering conditions, while producing small reprojection errors (Figs. 3, 7, 10, 14, 17, 22, 25, 29; Tables 3, 8) and guaranteeing positive depth estimates all the time. The way the proposed algorithm proves its tolerance on high percentage of outliers is not by detecting or removing outliers before computing the pose. In fact, without any outlier rejection scheme, the proposed algorithm demonstrates its tolerance on outliers. Of course, outliers can be screened and removed before algorithm execution, and outlier rejection scheme can be integrated as part of the algorithm. However, we are demonstrating the algorithm’s outlier tolerance capabilities itself, without extra techniques that help handle outliers, which often take place “outside” the algorithm. Many PnP algorithms evaluate their performance on degenerate configurations such as planar or quasi-singular configurations. We studied previous works that include these configurations and found that RPnP is robust in both planar and quasi-singular point configurations, it can also deal with non-planar configurations; EPnP struggles in co-planar configurations; and LHM assumes coplanar or weak-perspective point configurations, which may limit its wide adoption. For our algorithm, we randomly generated the data points in 3D space for simulation. Thus, they are not intentionally configured to be in co-planar, nor in quasi-singular configurations. We ran simulations that had points all on the same plane parallel to the image plane, and found that our algorithm produced inconsistent results only when the points were too close to the camera. If we made sure the distance from the camera to the object was at least half the length of the data span, the algorithm always produced consistent results. From one aspect, this finding proves our algorithm’s capability of handling co-planar configuration. In real-world scenarios, points used in the rigid box case of this paper were deliberately selected to avoid degenerate configurations such as co-planar. That is, most of the points were selected to be the box corners. In addition, our algorithm was also able to handle co-planar configuration in the satellite mockup scenario. Thus, through both simulation and real-world testing, our algorithm was able to handle certain degenerate configurations. In many of our experiments, we are aware that the number of points used starts from 10; while for many PnP algorithms, this number is valid if it is greater than 3. In addition to the family of PnP algorithms, since we are also testing the 5-pt algorithm, as well as the 8-pt algorithm, which require at least 5 or 8 points, to meet the point number requirement of all the algorithms, at least 8 points are needed. When varying the number of points, we could certainly choose the starting number to be 8. In fact, when setting the starting number to be 10 instead, it is consistent with the real-world scenarios of the study, where 10 control points were selected. Yet, in the experiment of “varying the number of points”, we chose the starting number to be 8 (8 to 30 points with an increment of 1). Mathematically, our Pos-dep (Min-eig-Depths) algorithm finds the depth as an eigenvector of a constructed data matrix. That is, instead of directly solving for rotation matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R$$\end{document}R and translation vector \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{t}}$$\end{document}t, it first finds all the “positive” eigenvectors (eigenvectors with all positive entries) of a data matrix, and picks the “positive eigenvector” associated with the smallest eigenvalue to be the depth vector, thus all depth values are guaranteed to be positive. To guarantee such “positive eigenvector” exists, the algorithm keeps taking randomly generated rotations as new initial guesses, until an eigenvector with all positive entries could be found. The number controlling the repetition times is a user-defined parameter. During our study, we have not encountered a case where no positive solutions could be found. However, to handle such rare case, we brute force set the resulting eigenvector with all entries equal to 1, which is a “positive” vector. Hence, the existence of such “positive” vector is proven empirically, rather than mathematically and we acknowledge that the existence of the “positive” eigenvector is achieved through empirical observation. After the “positive” eigenvector is found, the algorithm then uses Optimal Quaternion Algorithm41–46 to calculate a new rotation, and iteratively compares the current rotation with the new rotation under some set threshold (for example, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$5^\circ $$\end{document}5∘), then update \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{R}}$$\end{document}R if necessary (see Algorithm 1 in Methods). After the rotation is solved, the algorithm uses the solved \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{R}}$$\end{document}R, the solved (positive) depths and the input data (observations) to calculate the translation vector \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{t}}$$\end{document}t. As for computational cost and efficiency, the Pos-dep/Min-eig-Depths had a runtime of less than 35 ms when the number of points was less than 30 (Fig. 20). In real-world tests, the Pos-dep reported a runtime of about 6 ms (Table 5) and 4 ms (Table 10) on the two standard datasets, and a runtime of about 10 ms on the rigid box and the satellite mockup cases. Since the time-consuming sign check and solution rejection processes have been integrated into the Pos-dep itself, the algorithm automatically handles such processes without manual and standalone checks after algorithm execution. Therefore, no runtime is wasted on the cheirality check process after the pose is solved, which has already been taken care of at the algorithm execution phase. Although such mechanism may inherently increase the runtime of the algorithm itself, it incorporates the separate sign-checking process into algorithm execution altogether, circumventing extra workloads. There exist some limitations in terms of accuracy and runtime of the Pos-dep. As for accuracy, we do acknowledge that the proposed algorithm is less accurate than certain state-of-the-art (SOTA) algorithms as reported in Fig. 22. However, we found in Fig. 22a, when the point number reaches 13, the Pos-dep starts to yield a mean rotation error less than 5 degrees; in Fig. 22b, when the point number reaches 11, the median rotation error also starts to fall below 5 degrees, which is considered the threshold of good rotation estimate in this study. In Fig. 22c, when the point number reaches 13, the Pos-dep starts to yield a mean translation error less than 10%; in Fig. 22d, when the point number reaches 10, the median translation error also starts to fall below 10%, which is considered the threshold of good translation estimate in this study, and also applies to depth estimates. For reprojection errors, it can be seen from Fig. 22e and f that the Pos-dep was among the best, and when the point number reaches 14 and beyond, the Pos-dep reported an averaged reprojection error of less than 0.02 at each point number. In fact, one of the strengths of our algorithm is its ability of providing pose estimates with a small reprojection error, which is a crucial factor in pose estimation accuracy. It is also worth noting that Pos-dep performed less accurate (than EPnP) in Fig. 22, which is “varying the number of points” and is only one of the many experiments conducted in this work. In the experiment of “varying the percentage of outliers”, we can see from Fig. 14 that Pos-dep outperformed EPnP instead, on top of other benefits such as much smaller reprojection errors and positive depth estimates. It can be observed from the many experiments conducted in this paper that the Pos-dep may be less performant in one experiment, yet it can be more performant and even outperforms the most recent algorithms in another experiment (eg. Figures 15, 16, 17, 18, the Dino and the Temple datasets); and being able to constantly generate small reprojection errors with positive depth estimates, is a key advantage of the Pos-dep algorithm. As for runtime, indeed, when the number of points \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n$$\end{document}n = 30, the Pos-dep costs more time than other algorithms (Fig. 20). The increase on runtime with the increasing number of points is due to the increase of the dimension of the data matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{D}}}_{{\varvec{R}}}$$\end{document}DR, which is of size \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$2n\times 2n$$\end{document}2n×2n. We also believe that the runtime inefficiency is not due to factors like MATLAB. The OPnP is also originally implemented in MATLAB and its runtime is around 20 ms. In our experiments, we found the runtime of OPnP on our machine, which is also implemented in MATLAB, is around 8 ms when \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n$$\end{document}n < 30 (Fig. 20); and around 20 ms when \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n$$\end{document}n ranges from 100 to 1000 (Fig. 21b). There is practically no significant runtime increase when using MATLAB on different machines. Therefore, we think the runtime inefficiency is due to the algorithm itself, which tries to find the eigenvector corresponding to the minimal eigenvalue from a data matrix of size \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$2n\times 2n$$\end{document}2n×2n. The computation cost of eigenvector computation will then increase as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n$$\end{document}n goes up. We acknowledge that as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n$$\end{document}n increases, the Pos-dep may not be suitable for certain real-time applications, and it can be used as an offline backup solution to verify the results, particularly the results on reprojections. In the near future, we are aiming to improve the algorithm’s efficiency, possibly using parallelized programming or other multi-threaded acceleration techniques. There also exist some limitations on the algorithms and datasets used in this study. In terms of algorithms, there exist some recent algorithms such as Uncertain-PnP or PnP(L)26 and QPEPs27. The proposed PnP(L) method integrates the feature uncertainty with “globally convergent PnP(L) solvers, leveraging a complete set of 2D and 3D uncertainties”26 for pose estimation. The QPEPs method studies many quadratic pose estimation problems (QPEPs) including PnP, hand–eye calibration, point-to-plane registration, etc.27. This work proposed a “general quaternion-based mathematical model”27 to unify these QPEPs. Though both algorithms have MATLAB implementations, we tried to incorporate the PnP(L) into our study, while encountered difficulties when adapting its “method_list” parameter and “run_pnpl_method()” function into our configuration. We also tried to incorporate the QPEPs method into our study. We managed to replicate some test results presented in its paper, while had trouble in interpreting the quaternions and correlating its implementations in the “test_rel_att.m” and “test_stewart.m” scripts with our configuration. As for other recent algorithms such as CPnP and SQPnP, we found their implementations were more straightforward to port into our configuration and managed to incorporate them into this paper. In the near future, we are adding more algorithms proposed recently into our comparisons to keep our work up-to-date, starting with the PnP(L) and the QPEPs that already have MATLAB implementations. In terms of datasets, currently there exist more realistic computer vision datasets that are more suitable for real-world applications. Examples of these datasets and benchmarks include the KITTI Odometry Dataset37, the Robust Vision Challenge38, the ETH3D 2-view stereo benchmark39, the Heidelberg HD1K Stereo benchmark40, etc. For instance, the KITTI Odometry Dataset includes tasks of stereo, optical flow, visual odometry, 3D objects detection and 3D tracking. The dataset complements established datasets such as Middlebury by providing real-world benchmarks that can better reflect out-of-laboratory scenarios. Yet, in our work, we tested the algorithms using simulated data and four scenarios of real-world datasets. The first and the second scenarios are the Middlebury datasets, and the third and the fourth scenarios are self-constructed datasets. We acknowledge that to make the results more convincing, experiments with the datasets and benchmarks that better reflect real-world scenarios can be conducted. In the near future, we are expanding both the width and depth of our work by using these more realistic datasets. Despite the limitations discussed above, we believe one of the most significant contributions of the proposed Pos-dep algorithm is its neat yet elegant mathematical derivation process. Pos-dep tries to minimize the spatial translation error as a least squares problem. It uses the mathematical fact that the minimum of the least squares problem will be solved when the sample mean is achieved. In light of the Min-eig algorithm that treats the translation as an eigenvector, we treat the depth vectors as eigenvectors. Using the knowledge of minimization with constraints, we use Lagrange multiplier to formulate an eigenvalue solution. The problem then transforms to minimizing the minimum eigenvalue of a constructed data matrix whose dimension is solely dependent on the number of correspondence points. Next, the algorithm iteratively solves depth and rotation until some threshold is met. Finally, translation is estimated using the estimated depth and rotation. Details on this mathematical derivation process can be found in Methods. Methods To fully understand the technical details of our Pos-dep algorithm, we provide a complete mathematical derivation of our Pos-dep (Min-eig-Depths) algorithm from scratch, providing a mathematical foundation for the proposed algorithm. We will encounter multiple minimization problems during the following derivation process. We separate the whole process into three stages. The first stage aims to get rid of translation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{t}}$$\end{document}t from the problem, reducing the number of variables to be estimated. It starts from Eq. (3) with a rewritten form in Eq. (7). Taking \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{t}}$$\end{document}t as the only variable, we would be able to express \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{t}}$$\end{document}t using rotation and depths shown in Eq. (8). Hence, the problem onward only needs to control two variables: the rotation and the depths. The second stage is Eq. (25), which estimates depths as the eigenvector corresponding to the minimum eigenvalue. By providing an initial guess for the rotation, the only variable becomes depth. The final stage is Eq. (34), where the rotation is the only variable to be estimated. After the rotation and the depths are estimated, the translation is then calculated using Eq. (8) in a closed-form. Each of these three stages tackles only one variable at a time, after executing these stages, the depth, the rotation, and the translation can be solved in a sequential manner. We will demonstrate the mathematical derivation from formulating error function to the minimization methods, the method to obtain positive depth estimates, and how to later estimate rotation and translation using these depth estimates. Formulating error function J Assume the number of correspondence points between two images is n. Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{X}}}_{1}^{{\varvec{i}}}$$\end{document}X1i, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{X}}}_{2}^{{\varvec{i}}}$$\end{document}X2i (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i=1, 2, 3,\dots ,n$$\end{document}i=1,2,3,⋯,n) denote the 3D position of the ith point seen from the two camera coordinate systems, respectively; let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{R}}$$\end{document}R and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{t}}$$\end{document}t denote the rotation and translation from the first camera coordinate system to the second. Hence, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{X}}}_{1}^{{\varvec{i}}}$$\end{document}X1i, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{X}}}_{2}^{{\varvec{i}}}$$\end{document}X2i and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{t}}$$\end{document}t are 3 × 1 real vectors, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{R}}\in {\text{SO}}\left(3\right)$$\end{document}R∈SO3 is a rotation matrix. Then we have the following well-known equation for coordinate transform1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{X}}}_{2}^{{\varvec{i}}} = {\varvec{R}}{{\varvec{X}}}_{1}^{{\varvec{i}}}+{\varvec{t}}$$\end{document}X2i=RX1i+t Re-arranging Eq. (1) yields2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0=\left({{\varvec{X}}}_{2}^{{\varvec{i}}}-{\varvec{R}}{{\varvec{X}}}_{1}^{{\varvec{i}}}\right)-{\varvec{t}}.$$\end{document}0=X2i-RX1i-t. In real-world scenarios, due to the existence of noise, Eq. (2) does not equal the exact 0 vector. Therefore, we want to minimize the squared 2-norm of the right-hand side of Eq. (2) to get as close to 0 vector as possible. Since there are \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n$$\end{document}n correspondence points and the 2-norms are all positive, we can instead minimize the sum of these squared 2-norms over all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n$$\end{document}n points.3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\underset{{\varvec{t}}}{\mathit{min}}\sum_{i=1}^{n}{\Vert \left({{\varvec{X}}}_{2}^{{\varvec{i}}}-{\varvec{R}}{{\varvec{X}}}_{1}^{{\varvec{i}}}\right)-{\varvec{t}}\Vert }_{2}^{2},$$\end{document}mint∑i=1n‖X2i-RX1i-t‖22,where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{X}}}_{1}^{{\varvec{i}}}$$\end{document}X1i, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{X}}}_{2}^{{\varvec{i}}}$$\end{document}X2i again denote the 3D positions of the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i$$\end{document}i th point seen from the two camera coordinate systems. Now let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{x}}}_{1}^{{\varvec{i}}}$$\end{document}x1i, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{x}}}_{2}^{{\varvec{i}}}$$\end{document}x2i (3 × 1 vectors) denote the calibrated (homogeneous) image coordinates (last entry scaled to 1) of the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i$$\end{document}i th point on the first and second image plane; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\lambda }_{1}^{i}$$\end{document}λ1i and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\lambda }_{1}^{i}$$\end{document}λ1i (scalars) denote the depths of the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i$$\end{document}i th point to the two camera’s image planes, we have the following relationships between the 3D coordinates and the calibrated image coordinates of each camera4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{X}}}_{1}^{{\varvec{i}}}={\lambda }_{1}^{i}{{\varvec{x}}}_{1}^{{\varvec{i}}}$$\end{document}X1i=λ1ix1i5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{X}}}_{2}^{{\varvec{i}}}={\lambda }_{2}^{i}{{\varvec{x}}}_{2}^{{\varvec{i}}}.$$\end{document}X2i=λ2ix2i. Then Eq. (1) can be re-written as6\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\lambda }_{2}^{i}{{\varvec{x}}}_{2}^{{\varvec{i}}} = {\varvec{R}}{\lambda }_{1}^{i}{{\varvec{x}}}_{1}^{{\varvec{i}}}+{\varvec{t}}.$$\end{document}λ2ix2i=Rλ1ix1i+t. Substituting \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{X}}}_{1}^{{\varvec{i}}}$$\end{document}X1i and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{X}}}_{2}^{{\varvec{i}}}$$\end{document}X2i in Eq. (4) and Eq. (5) for those in Eq. (3), we have7\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\underset{{\varvec{t}}}{\mathit{min}}\sum_{i=1}^{n}{\Vert \left({\lambda }_{2}^{i}{{\varvec{x}}}_{2}^{{\varvec{i}}}-{\varvec{R}}{\lambda }_{1}^{i}{{\varvec{x}}}_{1}^{{\varvec{i}}}\right)-{\varvec{t}}\Vert }_{2}^{2},$$\end{document}mint∑i=1n‖λ2ix2i-Rλ1ix1i-t‖22,which is a least-squares problem. The least-squares estimate of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{t}}$$\end{document}t in Eq. (7) is the following sample mean (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$j=1, 2, 3,\dots ,n$$\end{document}j=1,2,3,⋯,n)8\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{t}}=\frac{1}{n}\sum_{j=1}^{n}{(\lambda }_{2}^{j}{{\varvec{x}}}_{2}^{{\varvec{j}}}-{\varvec{R}}{\lambda }_{1}^{j}{{\varvec{x}}}_{1}^{{\varvec{j}}}).$$\end{document}t=1n∑j=1n(λ2jx2j-Rλ1jx1j). Now Substituting the expression of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{t}}$$\end{document}t in Eq. (8) for that in Eq. (6), we have9\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\lambda }_{2}^{i}{{\varvec{x}}}_{2}^{{\varvec{i}}} = {\varvec{R}}{\lambda }_{1}^{i}{{\varvec{x}}}_{1}^{{\varvec{i}}}+\frac{1}{n}\sum_{j=1}^{n}{(\lambda }_{2}^{j}{{\varvec{x}}}_{2}^{{\varvec{j}}}-{\varvec{R}}{\lambda }_{1}^{j}{{\varvec{x}}}_{1}^{{\varvec{j}}}) .$$\end{document}λ2ix2i=Rλ1ix1i+1n∑j=1n(λ2jx2j-Rλ1jx1j). Let the right-hand side minus the left-hand side of Eq. (9) and call the difference as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${e}^{i}$$\end{document}ei. Also due to noise, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${e}^{i}$$\end{document}ei does not equal the exact zero vector \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0$$\end{document}0. The objective is to minimize the following \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${e}^{i}$$\end{document}ei, or \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${ne}^{i}$$\end{document}nei10\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${e}^{i} = {\varvec{R}}{\lambda }_{1}^{i}{{\varvec{x}}}_{1}^{{\varvec{i}}}+\frac{1}{n}\sum_{j=1}^{n}{(\lambda }_{2}^{j}{{\varvec{x}}}_{2}^{{\varvec{j}}}-{\varvec{R}}{\lambda }_{1}^{j}{{\varvec{x}}}_{1}^{{\varvec{j}}})-{\lambda }_{2}^{i}{{\varvec{x}}}_{2}^{{\varvec{i}}}$$\end{document}ei=Rλ1ix1i+1n∑j=1n(λ2jx2j-Rλ1jx1j)-λ2ix2i11\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${ne}^{i} = n{\varvec{R}}{\lambda }_{1}^{i}{{\varvec{x}}}_{1}^{{\varvec{i}}}+\sum_{j=1}^{n}{(\lambda }_{2}^{j}{{\varvec{x}}}_{2}^{{\varvec{j}}}-{\varvec{R}}{\lambda }_{1}^{j}{{\varvec{x}}}_{1}^{{\varvec{j}}})-n{\lambda }_{2}^{i}{{\varvec{x}}}_{2}^{{\varvec{i}}}.$$\end{document}nei=nRλ1ix1i+∑j=1n(λ2jx2j-Rλ1jx1j)-nλ2ix2i. At this stage let’s re-format the input data \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{x}}}_{1}^{{\varvec{i}}}$$\end{document}x1i and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{x}}}_{2}^{{\varvec{i}}}$$\end{document}x2i to form two new matrices \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{\chi}}}_{1}^{{\varvec{i}}}$$\end{document}χ1i and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{\chi}}}_{2}^{{\varvec{i}}}$$\end{document}χ2i12\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{\chi}}}_{1}^{{\varvec{i}}}=\left[{{\varvec{x}}}_{1}^{1},\boldsymbol{ }{{\varvec{x}}}_{1}^{2},\boldsymbol{ }{{\varvec{x}}}_{1}^{3},\boldsymbol{ }\dots ,\boldsymbol{ }{{\varvec{x}}}_{1}^{{\varvec{i}}-1}, \left(1-n\right){{\varvec{x}}}_{1}^{{\varvec{i}}},\boldsymbol{ }{{\varvec{x}}}_{1}^{{\varvec{i}}+1},\boldsymbol{ }\dots ,\boldsymbol{ }{{\varvec{x}}}_{1}^{{\varvec{n}}}\right]$$\end{document}χ1i=x11,x12,x13,⋯,x1i-1,1-nx1i,x1i+1,⋯,x1n13\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{\chi}}}_{2}^{{\varvec{i}}}=\left[{{\varvec{x}}}_{2}^{1}, {{\varvec{x}}}_{2}^{2}, {{\varvec{x}}}_{2}^{3}, \dots , {{\varvec{x}}}_{2}^{{\varvec{i}}-1}, \left(1-n\right){{\varvec{x}}}_{2}^{{\varvec{i}}}, {{\varvec{x}}}_{2}^{{\varvec{i}}+1}, \dots , {{\varvec{x}}}_{2}^{{\varvec{n}}}\right].$$\end{document}χ2i=x21,x22,x23,⋯,x2i-1,1-nx2i,x2i+1,⋯,x2n. Then, let14\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overrightarrow{{\lambda }_{1}}=\left[\begin{array}{c}{\lambda }_{1}^{1}\\ {\lambda }_{1}^{2}\\ {\lambda }_{1}^{3}\\ \vdots \\ {\lambda }_{1}^{n}\end{array}\right]$$\end{document}λ1→=λ11λ12λ13⋮λ1n15\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overrightarrow{{\lambda }_{2}}=\left[\begin{array}{c}{\lambda }_{2}^{1}\\ {\lambda }_{2}^{2}\\ {\lambda }_{2}^{3}\\ \vdots \\ {\lambda }_{2}^{n}\end{array}\right]$$\end{document}λ2→=λ21λ22λ23⋮λ2n Be the two \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n\times 1$$\end{document}n×1 vectors containing the depths. Then Eq. (11) can be written in a much more compact form16\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${ne}^{i} = {{\varvec{\chi}}}_{2}^{{\varvec{i}}}\overrightarrow{{\lambda }_{2}}-{\varvec{R}}{{\varvec{\chi}}}_{1}^{{\varvec{i}}}\overrightarrow{{\lambda }_{1}},$$\end{document}nei=χ2iλ2→-Rχ1iλ1→,or in a matrix form17\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${ne}^{i} = \left[\begin{array}{cc}-{\varvec{R}}{{\varvec{\chi}}}_{1}^{{\varvec{i}}}& {{\varvec{\chi}}}_{2}^{{\varvec{i}}}\end{array}\right]\left[\begin{array}{c}\overrightarrow{{\lambda }_{1}}\\ \overrightarrow{{\lambda }_{2}}\end{array}\right].$$\end{document}nei=-Rχ1iχ2iλ1→λ2→. Let18\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overrightarrow{\lambda }=\left[\begin{array}{c}\overrightarrow{{\lambda }_{1}}\\ \overrightarrow{{\lambda }_{2}}\end{array}\right],$$\end{document}λ→=λ1→λ2→,then from Eq. (17) we have19\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${ne}^{i} = \left[\begin{array}{cc}-{\varvec{R}}{{\varvec{\chi}}}_{1}^{{\varvec{i}}}& {{\varvec{\chi}}}_{2}^{{\varvec{i}}}\end{array}\right]\overrightarrow{\lambda }.$$\end{document}nei=-Rχ1iχ2iλ→. Since \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${ne}^{i}$$\end{document}nei is a 3 × 1 vector, to “minimize” \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${ne}^{i}$$\end{document}nei, let’s minimize its 2-norm as the following expression20\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\Vert {ne}^{i}\Vert }_{2}^{2}={n}^{2}{e}^{{i}^{T}}{e}^{i}={\overrightarrow{\lambda }}^{T}\left[\begin{array}{c}-{{{\varvec{\chi}}}_{1}^{{\varvec{i}}}}^{T}{{\varvec{R}}}^{T}\\ {{{\varvec{\chi}}}_{2}^{{\varvec{i}}}}^{T}\end{array}\right]\left[\begin{array}{cc}-{\varvec{R}}{{\varvec{\chi}}}_{1}^{{\varvec{i}}}& {{\varvec{\chi}}}_{2}^{{\varvec{i}}}\end{array}\right]\overrightarrow{\lambda },$$\end{document}‖nei‖22=n2eiTei=λ→T-χ1iTRTχ2iT-Rχ1iχ2iλ→,where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${(...)}^{T}$$\end{document}(...)T denotes the transpose operation. So far, we can introduce our error function J as21\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$J={n}^{2}\sum_{i=1}^{n}{e}^{{i}^{T}}{e}^{i}={\overrightarrow{\lambda }}^{T}(\sum_{i=1}^{n}{M}^{i}({\varvec{R}}))\overrightarrow{\lambda },$$\end{document}J=n2∑i=1neiTei=λ→T(∑i=1nMi(R))λ→,where22\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${M}^{i}\left({\varvec{R}}\right)= \left[\begin{array}{c}-{{{\varvec{\chi}}}_{1}^{{\varvec{i}}}}^{T}{{\varvec{R}}}^{T}\\ {{{\varvec{\chi}}}_{2}^{{\varvec{i}}}}^{T}\end{array}\right]\left[\begin{array}{cc}-{\varvec{R}}{{\varvec{\chi}}}_{1}^{{\varvec{i}}}& {{\varvec{\chi}}}_{2}^{{\varvec{i}}}\end{array}\right],$$\end{document}MiR=-χ1iTRTχ2iT-Rχ1iχ2i,according to Eq. (20). Next, let23\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{D}}}_{{\varvec{R}}}=\sum_{i=1}^{n}{M}^{i}({\varvec{R}}).$$\end{document}DR=∑i=1nMi(R). Then, from Eq. (21) we have24\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$J={\overrightarrow{\lambda }}^{T}{{\varvec{D}}}_{{\varvec{R}}}\overrightarrow{\lambda },$$\end{document}J=λ→TDRλ→,which is the formulated error function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$J$$\end{document}J. Note that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{D}}}_{{\varvec{R}}}$$\end{document}DR is a function of the unknown rotation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{R}}$$\end{document}R and the known image positions \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{x}}}_{1}^{{\varvec{i}}}$$\end{document}x1i and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{x}}}_{2}^{{\varvec{i}}}$$\end{document}x2i, where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i=1, 2, 3,\dots ,n$$\end{document}i=1,2,3,⋯,n. Minimizing J with positive depths constraint Now that the error function is obtained, the Min-eig algorithm searches its minimum. The Min-eig algorithm normalizes the translation vector \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{t}}$$\end{document}t to unit vector since there is inherent unavoidable ambiguity in the stereo image problem’s scale. Likewise, here we instead normalize the depths vector \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overrightarrow{\lambda }$$\end{document}λ→ to control the scale ambiguity. The other constraint of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overrightarrow{\lambda }$$\end{document}λ→ is that all its elements must be positive (positive depths constraint). The minimization problem can now be stated as\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\underset{\overrightarrow{\lambda } }{\mathit{min}}{J=\overrightarrow{\lambda }}^{T}{{\varvec{D}}}_{{\varvec{R}}}\overrightarrow{\lambda }$$\end{document}minλ→J=λ→TDRλ→subject to25\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Vert \overrightarrow{\lambda }\Vert =1=\sqrt{{\overrightarrow{\lambda }}^{T}\overrightarrow{\lambda } }={\overrightarrow{\lambda }}^{T}\overrightarrow{\lambda }, \overrightarrow{\lambda }>0,$$\end{document}‖λ→‖=1=λ→Tλ→=λ→Tλ→,λ→>0,where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overrightarrow{\lambda }>0$$\end{document}λ→>0 indicates all elements of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overrightarrow{\lambda }$$\end{document}λ→ are positive. Such constraint has been considered in our MATLAB code, guaranteeing that all entries in the eigenvector were either all positive or all negative. If all entries were negative, we would negate them to get all-positive entries. Next, for the other constraint regarding the norm, apply Lagrange multiplier \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu $$\end{document}μ and Lagrange function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal{L}$$\end{document}L:26\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal{L}=J+\mu (1-{\overrightarrow{\lambda }}^{T}\overrightarrow{\lambda } )$$\end{document}L=J+μ(1-λ→Tλ→)27\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\frac{\partial \mathcal{L}}{\partial \overrightarrow{\lambda } }=0=2{{\varvec{D}}}_{{\varvec{R}}}\overrightarrow{\lambda }-2\mu \overrightarrow{\lambda },$$\end{document}∂L∂λ→=0=2DRλ→-2μλ→,then28\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{D}}}_{{\varvec{R}}}\overrightarrow{\lambda }=\mu \overrightarrow{\lambda }.$$\end{document}DRλ→=μλ→. Rewriting error function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$J$$\end{document}J in Eq. (25) we can find the minimum of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$J$$\end{document}J as29\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${J}_{min}={\overrightarrow{\lambda }}^{T}\mu \overrightarrow{\lambda }=\mu {\overrightarrow{\lambda }}^{T}\overrightarrow{\lambda }=\mu .$$\end{document}Jmin=λ→Tμλ→=μλ→Tλ→=μ. From Eq. (28) we see that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu $$\end{document}μ is an eigenvalue of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{D}}}_{{\varvec{R}}}$$\end{document}DR (a \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$2n\times 2n$$\end{document}2n×2n matrix) and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overrightarrow{\lambda }$$\end{document}λ→ is the associated (unit) eigenvector. From Eq. (29) we can conclude that to minimize the error function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$J$$\end{document}J, we just need to minimize the minimum eigenvalue of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{D}}}_{{\varvec{R}}}$$\end{document}DR, with the associated eigenvector as the depths vector containing all positive entries. To fulfill the positive depths constraint, we need to first find all eigenvectors with all positive entries, then among them find the eigenvector associated with the smallest eigenvalue. Estimating rotation and translation from depths After we find the positive depths vector \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overrightarrow{\lambda }$$\end{document}λ→, we can estimate the pose using the input data and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overrightarrow{\lambda }$$\end{document}λ→. We used the Optimal Quaternion Algorithm39–44 (OQA, also see page 427 in Ref.27) to solve for rotation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{R}}$$\end{document}R as follows: Recall that Eq. (16) is30\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${ne}^{i} = {{\varvec{\chi}}}_{2}^{{\varvec{i}}}\overrightarrow{{\lambda }_{2}}-{\varvec{R}}{{\varvec{\chi}}}_{1}^{{\varvec{i}}}\overrightarrow{{\lambda }_{1}}.$$\end{document}nei=χ2iλ2→-Rχ1iλ1→. Let31\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{D}}}_{1}={{\varvec{\chi}}}_{1}^{{\varvec{i}}}\overrightarrow{{\lambda }_{1}}$$\end{document}D1=χ1iλ1→32\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{D}}}_{2}={{\varvec{\chi}}}_{2}^{{\varvec{i}}}\overrightarrow{{\lambda }_{2}},$$\end{document}D2=χ2iλ2→,be the inputs to the OQA, which are known to be dependent upon the transformed data \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{\chi}}}_{1}^{{\varvec{i}}}$$\end{document}χ1i, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{\chi}}}_{2}^{{\varvec{i}}}$$\end{document}χ2i, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overrightarrow{\lambda }$$\end{document}λ→, Eq. (30) now becomes33\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${ne}^{i} = {{\varvec{D}}}_{2}-{\varvec{R}}{{\varvec{D}}}_{1}.$$\end{document}nei=D2-RD1. Recalling Eq. (21), here, we want to minimize the sum of the squared 2-norm of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${ne}^{i}$$\end{document}nei over all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n$$\end{document}n points, that is (indices \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i=1, 2, 3,\dots ,n$$\end{document}i=1,2,3,⋯,n are implicit in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{D}}}_{1}$$\end{document}D1 and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{D}}}_{2}$$\end{document}D2),\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\underset{R}{\mathit{min}}\sum_{i=1}^{n}{\Vert {{\varvec{D}}}_{2}-{\varvec{R}}{{\varvec{D}}}_{1}\Vert }_{2}^{2}$$\end{document}minR∑i=1n‖D2-RD1‖22subject to 34\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{R}}\in SO\left(3\right).$$\end{document}R∈SO3. Given \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{D}}}_{1}$$\end{document}D1 and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{D}}}_{2}$$\end{document}D2, this problem can be analytically solved using the OQA. Recall Eq. (8), the translation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{t}}$$\end{document}t is then equal to35\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{t}}=\frac{1}{n}\sum_{j=1}^{n}{(\lambda }_{2}^{j}{{\varvec{x}}}_{2}^{{\varvec{j}}}-{\varvec{R}}{\lambda }_{1}^{j}{{\varvec{x}}}_{1}^{{\varvec{j}}}).$$\end{document}t=1n∑j=1n(λ2jx2j-Rλ1jx1j). Normalization of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{t}}$$\end{document}t gives us36\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{t}}=\frac{{\varvec{t}}}{\Vert {\varvec{t}}\Vert }.$$\end{document}t=t‖t‖. The positive depths values, are therefore also scaled by the norm of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{t}}$$\end{document}t37\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overrightarrow{{\lambda }_{1}}=\frac{\overrightarrow{{\lambda }_{1}}}{\Vert {\varvec{t}}\Vert }$$\end{document}λ1→=λ1→‖t‖38\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overrightarrow{{\lambda }_{2}}=\frac{\overrightarrow{{\lambda }_{2}}}{\Vert {\varvec{t}}\Vert }.$$\end{document}λ2→=λ2→‖t‖. The iterative nature of the proposal The rotation and depths are estimated iteratively. First, we provide an initial guess for rotation R and finds the eigenvector (depth vector) of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{D}}}_{{\varvec{R}}}$$\end{document}DR with all positive or negative entries whose associated eigenvalue is the smallest; second, use OQA to solve for a new rotation estimate Rnew. If the difference between the initial guess R and Rnew is within some threshold, the final estimated rotation is Rnew; otherwise, take Rnew as the new initial guess and repeat the above process, until the difference between the initial guess and Rnew is within some threshold. We can see rotation and depths are updated iteratively. The Pos-dep algorithm Based on the derivation above, we can summarize the proposed Pos-dep algorithm as follows. See Algorithm 1. Algorithm 1The Pos-dep Algorithm. The Pos-dep Algorithm. Generating problem instances For the simulation part, taking “varying the noise levels” as an example, we chose arbitrary calibration matrix with randomized rotation and translation; we also randomized 3D positions \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{X}}}_{1}^{{\varvec{i}}}$$\end{document}X1i and calculated \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{X}}}_{2}^{{\varvec{i}}}$$\end{document}X2i using Eq. (1); then we were able to generate calibrated and uncalibrated (pixel) coordinates, which are the inputs of various pose estimation algorithms. The way we added noise is as follows: we added random Gaussian noise with different noise standard deviations, known as “noise levels”, to the calibrated coordinates \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{x}}}_{1}^{{\varvec{i}}}$$\end{document}x1i, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\varvec{x}}}_{2}^{{\varvec{i}}}$$\end{document}x2i. For each noise level, 100 tests were performed to calculate the percentage of good estimates at that noise level. In addition, the mean and median estimation errors of the algorithms were also studied. For the real-world part, the correspondences were found using SIFT algorithm for the standard datasets, and were selected manually for the rigid box and the satellite mockup cases. The camera intrinsics in the standard datasets were available thus no calibration was needed. We self-calibrated the cameras using checkerboard patterns for the rigid box and the satellite mockup cases to obtain calibrated image coordinates. For the rigid box image and the satellite mockup image, the 3D positions of the points were manually measured using a ruler, and the 2D coordinates were obtained through MATLAB’s “getpts” control point selection, which were later converted to calibrated image coordinates using the calibration information. Integrating the SQPnP and CPnP algorithms The SQPnP algorithm has been implemented in OpenCV using the solvePnP function with the flag option set as SOLVEPNP_SQPNP. We used OpenCV’s Python package “cv2” and ported the Python script into MATLAB using the “py” prefix preceding the Python expressions. In this way, Python script of SQPnP can be directly used inside MATLAB and made comparable with other algorithms. The CPnP algorithm, like most other PnP algorithms used in this study, its MATLAB implementation is available online at: https://github.com/LIAS-CUHKSZ/CPnP-A-Consistent-PnP-Solver/tree/main/CPnP. Conclusion We proposed a novel camera pose estimation algorithm, the Pos-dep/Min-eig-Depths algorithm. The algorithm was formulated based on comprehensive mathematical derivation, which solves for depth as the eigenvector associated with the minimum eigenvalue of a constructed data matrix. The algorithm was tested along with many other pose estimation algorithms under various noisy conditions in numerous simulated and real-world experiments. The proposed algorithm showed better or comparable performance than the SOTA in certain experiments, while producing small reprojection errors and guaranteeing positive depths. In the near future, we are aiming to improve the accuracy and efficiency of the proposed algorithm. We will also incorporate more recently proposed algorithms, as well as more realistic computer vision datasets into our study. Supplementary Information Supplementary Information. Supplementary Information.