Mathematical Camera Array Optimization for Face 3D Modeling Application Authors: Bashar Alsadik, Luuk Spreeuwers, Farzaneh Dadrass Javan, Nahuel Manterola Published: 2023 ABSTRACT Camera network design is a challenging task for many applications in photogrammetry, biomedical engineering, robotics, and industrial metrology, among other fields. Many driving factors are found in the camera network design including the camera specifications, object of interest, and type of application. One of the interesting applications is 3D face modeling and recognition which involves recognizing an individual based on facial attributes derived from the constructed 3D model. Developers and researchers still face difficulty in reaching the required high level of accuracy and reliability needed for image-based 3D face models. This is caused among many factors by the hardware limitations and imperfection of the cameras and the lack of proficiency in designing the ideal camera-system configuration. Accordingly, for precise measurements, we still need engineering-based techniques to ascertain the specific level of deliverables quality. In this paper, an optimal geometric design methodology of the camera network is presented by investigating different multi-camera system configurations composed of four up to eight cameras. A mathematical nonlinear constrained optimization technique is applied to solve the problem and each camera system configuration is tested for a facial 3D model where a quality assessment is applied to conclude the best configuration. The optimal configuration is found to be a 7-camera array, comprising a pentagon shape enclosing two additional cameras, offering high accuracy. For those who prioritize point density, a 9-camera array with a pentagon and quadrilateral arrangement in the X-Z plane is a viable choice. However, a 5-camera array offers a balance between accuracy and the number of cameras. 1. Introduction Face recognition is a biometrics recognition approach that involves recognizing an individual based on their facial characteristics or features. Interestingly, 3D face recognition is a system that takes advantage of the human face’s 3D geometric information. It uses data from 3D sensors to determine the shape of a person’s face and to validate his/her stated identity by matching geometric features extracted from the 3D reconstructed faces to recognize people against a dataset. By utilizing features that are not susceptible to lighting conditions, head orientation, varying facial expressions, and makeup, 3D face recognition has the potential to reach a higher accuracy than its 2D equivalent [1]. However, collecting the 3D face depth data can be achieved either using active sensors or passive sensors. Active sensing techniques can be based on laser triangulation [2], structured light [3], and time-of-flight [4,5]. The structured light technology (Figure 1) is based on using speckle images with particular coding to determine depth but some concerns such as sensitivity to ambient illumination and occlusions are currently being researched. By using numerous cameras, laser triangulation achieves submillimeter precision and prevents occlusions. However, it claims to capture periods of many seconds, like many other laser triangulation systems, rendering it inaccurate for scanning the face of a moving person. Time-of-flight systems currently have insufficient accuracy and information density to be used reliably for 3D face recognition [6]. On the other hand, passive sensors for 3D face recognition are currently thriving because of the advancements in using deep learning to enable a reconstruction from a single 2D image [7,8]. Still, the stereo vision using the camera array is the most reliable for 3D face modeling and recognition since it provides a realistic occlusion-free model. However, such a camera array system requires a sophisticated design to enable highly accurate 3D face modeling. Table 1 summarizes the advantages and disadvantages of the mentioned remote sensing techniques for 3D modeling. Building a camera array system is necessary to ensure simultaneous capture of the face at the same instant to avoid any deformation in the reconstructed 3D model of such a nonrigid body. Therefore, further research on camera arrays designed for 3D face reconstruction is required to reach the high accuracy of active approaches as will be presented in this paper. Accordingly, the question that arises is how many cameras are enough for accurate facial 3D reconstruction? and in which reasonable configuration? Currently, several multiarray camera systems are designed for the gaming and film industry, textile industry, medical industry, etc. like the examples in [22,23]. However, those systems require high costs, and large space, and may not fit facial modeling which requires a focused camera system. Using images for precise measurements and 3D modeling is a major task in the fields of computer vision, photogrammetry, and robotics. In most of the mentioned image-based applications, it is required to have high geometric specifications including:Sufficient overlap percentage among an acceptable number of captured images.Suitable ray intersection geometry of the images defined by the base/height (B/H) ratio. The B/H ratio is an expression of the acceptable base distance B between the cameras themselves and the distance to the object H.Acceptable angles of incidence between the image rays and the object features.Pre-calibrated camera or pre-identified interior camera parameters. Sufficient overlap percentage among an acceptable number of captured images. Sufficient overlap percentage among an acceptable number of captured images. Suitable ray intersection geometry of the images defined by the base/height (B/H) ratio. The B/H ratio is an expression of the acceptable base distance B between the cameras themselves and the distance to the object H. Suitable ray intersection geometry of the images defined by the base/height (B/H) ratio. The B/H ratio is an expression of the acceptable base distance B between the cameras themselves and the distance to the object H. Acceptable angles of incidence between the image rays and the object features. Acceptable angles of incidence between the image rays and the object features. Pre-calibrated camera or pre-identified interior camera parameters. Pre-calibrated camera or pre-identified interior camera parameters. Moreover, achieving optimal results necessitates favourable imaging conditions during image captures, including adequate scene illumination, stable capture free from shaking, and effective occlusion avoidance. These specified conditions collectively define the parameters of an optimal camera network. The objective of having ideal or optimal camera networks is discussed several times in the literature [24,25,26,27]. The design task is aimed to be automatically applied to construct a robust network of overlapped images covering the required object and ensure reliability. The method of the ‘next best view’ NBV represents the famous approach for the strategy of growing a few images into many [28,29]. The NBV method assumes a robot that only knows the position and the approximate dimensions of the object in question. Accordingly, the NBV search-based method is applied by adding one view (camera) selected among a set of candidate views and should fulfil some constraints related to visibility, accessibility, angle of incidence, and overlap. This NBV approach is iteratively applied by adding new views while the robot is navigating. However, the NBV methods pay more attention to the uncertainty at the robot positioning waypoints compared to the uncertainty at the object in the question itself. Other research work was applied to find the ideal camera network based on the strategy of filtering many initial images to a minimum [30]. The filtering approach is based on an initial design of a very dense camera network around the rough point cloud of the object. This dens camera network is examined iteratively to indicate redundant images. In more detail, this filtering technique is based on the concept of having at least three images viewing the object points instantaneously. Hence, the redundant images are filtered out if they exclusively image only points that are covered by more than three cameras and then followed by an optimization step using the nonlinear-constrained minimization. In this paper, we aim to find the best configuration of a camera array system for 3D face modeling using optimization techniques and recommend the most suitable one. The paper sections are sequenced as follows: in Section 2, the proposed methodology will be explained in detail. In Section 3, 3D face modeling through optimized camera arrays will be presented. In Section 4, we will discuss the results and end up with the research conclusion. 2. Methodology As mentioned in the previous section, we propose a novel approach to a computerized camera network design that will conclude the optimal configuration of a camera array system for 3D face recognition. This will be applied by following the strategy of initializing a specific number of cameras and followed by mathematical optimization computations to fulfil the setup constraints against the required accuracy at the object space and stopping when required limits are reached. Therefore, a nonlinear constrained optimization starts from initial orientation values which are expected to converge rapidly to the global minimum solution. Figure 2 illustrates the proposed conceptual framework. The developed optimization workflow as shown in Figure 2 will be designed to minimize the total error in the object points (Section 2.3.1). However, different constraints must be satisfied during the optimization as will be shown in detail in Section 2.3.2. Mostly, the optimal camera network is constrained to different design requirements like the allowed B/H ratio which is highly contributing to getting an effective dense 3D reconstruction and accurate ray intersection. B/H is associated with the required ground sampling distance GSD, scale, camera angular field of view, and the required accuracy. It’s worth mentioning that the final aim of the imaging task will have a direct impact on the designed constraints of the optimization algorithm. For 3D face modeling, a short baseline network design is preferred where the B/H should be in the range of 15–30% [31,32]. On the other hand, wide baseline networks are designed for applications that require a high positional quality like structural deformation monitoring or laboratory camera calibration. Therefore, a B/H ratio of about 60% or greater is recommended. Another optimization constraint is the camera viewing angle or the angle of incidence which is of comparable importance to the B/H ratio as will be illustrated in Section 2.3.2. 2.1. Automated Initial Camera Network Design To design an initial camera network, it is efficient to downsample the dense point cloud of the object. A uniform sampling approach is applied by the division of the dense point cloud into a regular grid of voxels. The size of voxels is determined by a specified sampling density or point spacing. All points falling within each voxel are considered as one group. The average of point positions (or center of mass) of the points within each voxel is computed. Then downsampled point cloud is formed by using these average points within each voxel (Figure 3). This uniform sampling method is intended to reduce the density of point clouds while preserving the structure of the face. The rough point cloud of the object is then clustered into a specific number of clusters using k-means clustering where the points are partitioned in such a way that they are as close to one other as possible while being as far apart as possible from points in other clusters. This is done by minimizing the sum of distances between the cluster’s centroid and all of its points in each cluster. Accordingly, the total number of cameras required will be used to specify the number of point clusters. Then for every cluster of points, the mean normal direction is calculated to define the optical axis of a viewing camera at an initial distance. Then the vector direction of the camera optical axis is converted into the rotation matrix M to complete the set of the six exterior orientation elements of each viewing camera. To compute the rotation matrix M, first, we calculate the direction angle α of the initialized camera optical axis by using the dot product between two vectors a⃑ and N⃑ as in Equation (1):(1)cos⁡α=a⇀·N⇀ The angles between the initial camera axis and the adopted XYZ coordinate system can be calculated using the cross-product between the mentioned normalized vectors as in Equation (2):(2)θXYZ=a⃑×N⃑ The described geometry is shown in Figure 4 where a⃑: The camera orientation in a nadir viewing [00−1]. N⃑: The normal direction vector of a cluster [n1n2n3]. θXYZ: The angles enclosed between the normal vector and the three axes as [θxθyθz]. The initial camera rotation definition uses the normal pointing vector and Rodrigues rotation formula. Then, the rotation matrix defining the camera orientation in space can be derived using the following Equation (3) which is based on using Rodrigues’ Rotation formula [33,34]. (3)M=(1−cos α)θx2+cos α(1−cos α)θx θy−sin α θz(1−cos α)θx θz+sin α θy1−cos αθx θy+sin α θz(1−cos α)θy2+cos α(1−cos α)θy θz−sin α θx1−cos αθx θz−sin α θy(1−cos α)θy θz+sin α θx(1−cos α)θz2+cos α Figure 5 illustrates a four-camera network initialization using four clusters of points. 2.2. Elements of the Mathematical Optimization After the camera network initialization, optimization techniques will be followed. Optimization is generally formulated to compute a set of unknown parameters in a mathematical model x=(x1,x2,…xn) that can be defined as optimal. The optimization problem can be unconstrained in a simple case, this might be a minimization or a maximization problem. A more challenging optimization problem is found when the objective (cost) function f(x) to be minimized or maximized is subject to constraints in the form of equality constraints, hix=0 (i=1,….,me), inequality constraints, gix≤0 (i=me+1,…,m); and lower xl to upper xu parameter bounds. The solution of the nonlinear unconstrained minimization problem or nonlinear least-squares problem with redundant observations is either to be solved by Levenberg–Marquardt or by Gauss-Newton methods [35]. However, when the system of equations is constrained then it is harder to solve. Typically, the constrained minimization problem is solved by introducing the LaGrange multipliers λ composed of both quality λh and inequality constraints λg as follows in Equation (4):(4)Lx,λ=fx+∑λg,igi(x)+∑λh,ihi(x) The Karush-Kuhn-Tucker (KKT) conditions must be met to discover the optimal solution and ensure a global optimum for complicated minimization conditions [36]. It is worth mentioning that LaGrange multipliers λ convert the inequality constraints formulation into equality formulation in order to establish a stationary point where the partial derivatives are zero. As a result, in limited situations, generates a required condition for optimality. Solving a large-scale nonlinear constrained minimization problem, as in the case of camera network optimization, is a difficult task. Trust region, sequential quadratic programming (SQP), and interior-point algorithms can be employed to address nonlinear-constrained optimization problems [36,37,38]. According to the literature, the interior-point technique has had a lot of success and has proven to be useful for a wide range of problem classes because of its regularization effects on the constraints. Because of their Newton-like properties in terms of scalability and convergence performance, interior-point methods have become the trusted solution method for large-scale optimization problems, according to [39,40]. As a result, the interior-point optimization technique will be used to tackle the camera network optimization problem in this research study. 2.3. The Formulation of the Camera Network Optimization Problem The mathematical model that represents the core of the camera network design and relates the interior and exterior camera parameters to the object coordinates is the collinearity equations model as illustrated in Equation (5). It should be noted that the bundle adjustment method which is based on the collinearity equations is widely used when estimating the adjusted camera parameters and the object coordinates [41]. (5)FxA=−fm11Xj−Tx+m12Yj−Ty+m13Zj−Tzm31Xj−Tx+m32Yj−Ty+m33Zj−Tz−xFyA=−fm21Xj−Tx+m22Yj−Ty+m23Zj−Tzm31Xj−Tx+m32Yj−Ty+m33Zj−Tz−y where FxA and FyA represent the differences between the observed image coordinates x and y and their computed values. f: focal length. x,y: image coordinates. Tx,Ty,Tz: camera coordinates. Xj,Yj,Zj: object point coordinates. m’s: rotation matrix element derived from three angles ω,φ,k and based on a right-handed system. As mentioned in the previous section, the most costly computational step is to solve the large-scale mathematical constrained minimization problem especially if the 3D face is represented by a large number of n points and with a very small tolerance for stopping criterion. In summary, the optimization problem of the camera network design needs a precise definition of the input and output parameters which can be listed as follows: The input data parameters: Point coordinates defining the object (Xj,Yj,Zj,j=1:n). For every initial camera i, there are six initial exterior orientation parameters x0=ωi°,φi°,κi°,Txi°,Tyi°,Tzi°. The parameters vector x0 represents the initial guess of unknowns for running the subsequent optimization step. The output parameters: The optimal exterior orientation parameters x^=ω^i,φ^i,k^i,Tx^i,Ty^i,Tz^i for each designed camera i in the whole camera array network. It should be noted that between the mentioned input and output steps, there are many processing formulations regarding the cost function and the optimization constraints as will be discussed in the following sections. 2.3.1. Cost Function As mentioned, the objective of the optimization is to build a strong camera network that ensures minimum errors or higher accuracies at the object points. Accordingly, the cost function is formulated by computing the covariance matrix of the object points Qs using the least-squares adjustment method as shown in Equation (6) [21]. (6)Qs=BtWB−1=σX2σY2σZ2 where B: the matrix of the partial derivatives of the collinearity equations concerning the object coordinates (X,Y,Z). W: the weighting matrix. σX2,σY2,σZ2: variances at XYZ coordinates respectively. Accordingly, the cost function G is designed to minimize the norm of the eigenvalues (λ1, λ2,λ3) of the covariance matrix Qs as shown in Equation (7). (7)G=min⁡eigen Qs=minλ1, λ2,λ3 where | | refers to the norm. Since the eigenvalues represent the error ellipsoid axes lengths at each object point, this cost function of Equation (7) is meant to improve the accuracy of the whole camera network. As mentioned, the camera optimization problem is nonlinear and needs to be constrained to obtain realistic results that satisfy the final goal of the imaging. In the next Section 2.3.2, an explanation is given about the necessary constraints involved in camera network optimization. 2.3.2. Network Design Constraints The camera network design problem is influenced by specific geometric constraints, which can be listed as follows: The lower and upper bounds of the estimated parameters for each designed camera (Equation (8)) (8)−90°<ωi<90°−90°<φi<90°−180°0)=0 A final worthy to mention remark is that it can happen in the nonlinear-constrained minimization that the solution is well converged, and the step length is smaller than its threshold value while the constraints are not fully satisfied. We will consider this, if occurred, as an acceptable result since getting very close to the numbers we aimed for in the constraints meets our design requirements as well. 2.4. Pseudocode To summarize the proposed minimal optimal camera network design workflow, a pseudo-code is given in Algorithms 1–3 (Pseudocode: summary of the proposed minimal optimal camera network design workflow) as follows: Algorithm 1: Main program includes the input and output and call the optimization.functions of both: cost function and nonlinear constraints.Input:– object points P(X,Y,Z)– camera parameters: focal length, frame size, pixel size, lens distortion.– initial camera orientation ω°,φ°,k°,Tx°,Ty°,Tz° for 1:num. of camerascall Algorithm 2call Algorithm 3run nonlinear constrained minimzation using the interior-point method.Output: optimal camera orientation (ω^,φ^,k^,Tx^,Ty^,Tz^)Print results. Algorithm 2: Compute the cost function of minimizing the Q matrix of the object points.Input: initial camera orientation and parameters, object points P and their normal directions.Output: cost function F min.eigen (Q covariance matrix)For j = 1:PFor i = 1:no. of camerascompute rotation matrix M compute image coordinates.endcheck visibility of Pj in camera icompute covariance matrix Qjendcost function F = |eig(Q)| Algorithm 3: Compute the nonlinear constraints function of the camera design.Input: initial camera orientation and parameters, object points P and their normal directions.Output: nonlinear constraints [c,ceq]For j = 1:PFor i = 1:no. of cameras compute rotation matrix M. compute angle of incidence ij. compute image coordinates. endcheck visibility of Pj in camera iendFor h = 1:no. of camerascompute B/H rationonequality constraints c=abs(xpj)