PDGrad: Guiding Diffusion Model for Reference-Based Blind Face Restoration with Pivot Direction Gradient Guidance Authors: Geon Min, Tae Bok Lee, Yong Seok Heo Published: 2024 ABSTRACT Reference-based blind face restoration (RefBFR) has gained considerable attention because it utilizes additional reference images to restore facial images in situations where the degradation is caused by unknown factors, making it particularly useful in real-world applications. Recently, guided diffusion models have demonstrated exceptional performance in this task without requiring training. They achieve this by integrating gradients of the losses where each loss reflects the different desired properties of the additional external images. However, these approaches fail to consider potential conflicts between gradients of multiple losses, which can lead to sub-optimal results. To address this issue, we introduce Pivot Direction Gradient guidance (PDGrad), a novel gradient adjustment method for RefBFR within a guided diffusion framework. To this end, we first define the loss function based on both low-level and high-level features. For loss at each feature level, both the coarsely restored image and the reference image are fully integrated. In cases of conflicting gradients, a pivot gradient is established for each level and other gradients are aligned to it, ensuring that the strengths of both images are maximized. Additionally, if the magnitude of the adjusted gradient exceeds that of the pivot gradient, it is adaptively scaled according to the ratio between the two, placing greater emphasis on the pivot. Extensive experimental results on the CelebRef-HQ dataset show that the proposed PDGrad significantly outperforms competitive approaches both quantitatively and qualitatively. 1. Introduction Blind face restoration (BFR) aims to restore a high-quality (HQ) face image from a low-quality (LQ) image that has been degraded by unknown and complex factors, such as downsampling, blur, noise, and compression artifacts. BFR is a highly ill-posed problem because the unknown degradation makes it difficult to determine a single solution for a given LQ image, leading to multiple possible outcomes. Since facial images are sensitive to even subtle differences, having detailed information is essential for accurate restoration. By utilizing high-quality (HQ) reference images of the same individual, it becomes possible to achieve a high quality of image that is difficult to attain with BFR methods that do not use reference images. In this context, the reference-based blind face restoration (RefBFR) method has gained significant attention for its unique ability to leverage additional reference images to improve accuracy for practical scenarios. As a result, it can be applied for various applications, including face recognition [1,2], face detection [3,4,5] and age estimation [6,7]. Recently, several RefBFR studies [8,9,10,11,12,13] have been proposed based on deep learning [14]. Among these methods, PGDiff [13] has demonstrated outstanding performance in RefBFR using a training-free guided diffusion model [15]. It provides guidance to an unconditional diffusion model pre-trained for face image generation by incorporating the gradients of the losses during the reverse diffusion process. Their loss function is structured as a combination of multiple distances, each representing a specific desired attribute of the additional images. These include the coarsely restored image, generated using an external restorer such as CodeFormer [16], and the reference image, processed through the ArcFace network [1]. However, the guidance technique of PGDiff [13] may not be the optimal solution for RefBFR. This limitation arises from their gradients, which focus on using low-level information solely from the coarsely restored image, while relying on high-level information exclusively from the reference image. As a result, this approach fails to capture the crucial low-level details from the reference image and the high-level features from the coarsely restored image, leading to sub-optimal results. Moreover, the guidance derived from merely summing the gradients of multiple loss functions often results in sub-optimal results, as these gradients may be incompatible, causing conflicts. To address this problem, we propose a novel gradient adjustment method for RefBFR called Pivot Direction Gradient guidance (PDGrad) within a guided diffusion framework. Inspired by PCGrad [17], the essence of our method is to reduce gradient interference by directly modifying the conflicting gradients of the loss. To this end, we first define the loss function based on both low-level and high-level features. Similar to PGDiff [13], we utilize external information such as the coarsely restored image yc which is obtained using the pre-trained restoration method such as CodeFormer [16] and the reference yr. However, unlike PGDiff, we utilize both yc and yr to compute the loss at each level. This is because these two images capture complementary characteristics of face images. Figure 1 illustrates the complementary properties of yc and yr. Generally, yc is aligned well with the LQ input and making it easy to compare with the prediction for low-level information such as edge, color and shape. However, certain areas of yc are not restored effectively. In contrast, yr provides more reliable high-level information, such as identity, and is partially aligned to input, helping to compensate for the low-level details in regions where yc has significant degradation. Based on this observation, our approach efficiently and comprehensively leverages both images, enabling the effective integration of detailed and contextual information from both yc and yr. In this situation, simply summing the gradients of the losses at each level can lead to conflicting gradients. To address this issue, we establish a proper pivot gradient for the loss at each feature level and align other gradients to this pivot when conflicts arise. This approach allows us to fully harness the distinct advantages of both yc and yr. Specifically, for the loss using low-level features, the gradient from the loss using yc is prioritized, and the gradient of the loss using yr is modified by projecting it onto the plane orthogonal to yc when a conflict arises. Conversely, for the loss using high-level features, the gradient from the loss using yr is emphasized, and the gradient from the loss using yc is projected onto that of yr to avoid conflict and fully utilize the information in yr. Additionally, if the magnitude of the adjusted gradient exceeds that of the pivot gradient, it is adaptively scaled according to the ratio between the two, placing greater emphasis on the pivot. As exemplified in Figure 1, the proposed PDGrad outperforms previous methods by preserving the properties of the prioritized image at each feature level while selectively extracting elements of the properties of other images in a manner that aligns with the prioritized image. In summary, the proposed method provides the following key contributions:We propose a novel gradient adjustment method called PDGrad for RefBFR within a training-free guided diffusion framework.The loss function of the proposed method consists of two components: low-level and high-level losses, where both the coarsely restored image and the reference image are fully incorporated.Our proposed PDGrad establishes a proper pivot gradient for the loss at each level and adjusts other gradients to align with this pivot by modifying their direction and magnitude, thereby mitigating gradient interference.Extensive comparisons show the superiority of our method against previous state-of-the-art RefBFR methods. We propose a novel gradient adjustment method called PDGrad for RefBFR within a training-free guided diffusion framework. We propose a novel gradient adjustment method called PDGrad for RefBFR within a training-free guided diffusion framework. The loss function of the proposed method consists of two components: low-level and high-level losses, where both the coarsely restored image and the reference image are fully incorporated. The loss function of the proposed method consists of two components: low-level and high-level losses, where both the coarsely restored image and the reference image are fully incorporated. Our proposed PDGrad establishes a proper pivot gradient for the loss at each level and adjusts other gradients to align with this pivot by modifying their direction and magnitude, thereby mitigating gradient interference. Our proposed PDGrad establishes a proper pivot gradient for the loss at each level and adjusts other gradients to align with this pivot by modifying their direction and magnitude, thereby mitigating gradient interference. Extensive comparisons show the superiority of our method against previous state-of-the-art RefBFR methods. Extensive comparisons show the superiority of our method against previous state-of-the-art RefBFR methods. In this paper, we outline the organization as follows: Section 2 discusses previous works on blind face restoration. Section 3 provides detailed explanation of the proposed PDGrad. In Section 4, we compare and analyze the experimental outcomes of several methods, including our proposed approach. Finally, in Section 5, we discuss the conclusions. 2. Related Works Most recent BFR studies have focused on utilizing face-specific prior information, such as geometric facial priors, reference priors and generative facial priors. Note that our proposed method can be viewed as a study that exploits both reference priors and generative facial priors. Geometric Facial Priors. Geometric Facial Priors. Unlike natural images, faces consist of a common structural shape and components (e.g., eyes, nose, mouth and hair). Inspired by this, several approaches have been proposed to utilize the geometric priors. including facial landmarks [18,19], semantic segmentation map [20,21,22,23] and 3D shapes [24,25,26]. However, as pointed out by [16,27], such priors have limitations in guiding the fine details and texture information of the face (e.g., wrinkles and eye pupils). Furthermore, estimating geometric face priors from severely degraded inputs makes it difficult to obtain reliable results, which can affect performance. Reference Priors. Reference Priors. Various methods [8,9,11] have been developed to utilize high-quality facial images of the same individual as reference for restoration, aiming to leverage the distinct facial features of each person. However, these methods heavily rely on reference images of the same individual, which cannot be easily accessible. To mitigate this issue, DFDNet [10] utilizes a facial component dictionary as reference information. However, their approach may be sub-optimal for face restoration tasks since their dictionary is extracted from the pre-trained face recognition model. Inspired by [10], DMDNet [12] introduces dual dictionaries that extend beyond a single general dictionary, allowing for more flexible handling of degraded inputs, regardless of whether reference images are present. While [10,12] utilize the facial component dictionary extracted from the face recognition model, ENTED [28] introduces a vector quantized dictionary along with a latent space refinement technique. In contrast to the above methods that leverage a single reference image, ASFFNet [11] utilizes multiple reference images to select the most suitable guidance image and learns the landmark weights to improve the reconstruction quality. ENTED [28] is a blind face restoration framework that uses a high-quality reference image to restore a single degraded input image. It substitutes corrupted semantic features with high-quality codes, inspired by vector quantization, and generates style codes containing high-quality texture information. PFStorer [29] utilizes a diffusion model for face restoration and Super-Resolution, using several images of the individual’s face to customize the restoration process while preserving fine details. Generative Facial Priors. Generative Facial Priors. Recently, numerous studies have widely leveraged the capability of generative models such as the Generative Adversarial Network (GAN) [30], Vector Quantized-Variational AutoEncoder (VQVAE) [31], Vector Quantized-Generative Adversarial Network (VQGAN) [32] and Denoising diffusion probabilistic models (DDPMs) [33,34,35]. GAN inversion-based methods [36,37] try to find the closest latent vector in the GAN latent space corresponding to a given input image. GFP-GAN [27] and GPEN [38] design their encoder networks to effectively find the latent vector for an input image, then utilize the pre-trained GAN model as a decoder in their methods. VQFR [39] uses a vector-quantized (VQ) codebook as a dictionary to enhance high-quality facial details. By employing a parallel decoder for the fusion of input features with texture features from the VQ codebook, this approach preserves fidelity while achieving detailed facial restoration. CodeFormer [16] is a transformer-based architecture for code prediction that captures the global structure of low-quality facial images, enabling the generation of natural faces even from severely degraded inputs. To adapt to varying levels of degradation, a controllable feature transformation (CFT) module is included, offering a versatile balance between fidelity and quality. RestoreFormer++ [40] enhances facial image restoration by utilizing fully spatial and multi-head cross-attention to merge contextual, semantic and structural information from degraded face features with high-quality priors. PMRF [41] presents an algorithm that predicts the posterior mean and then uses a rectified flow model to transport it to a high-quality image. Building on the powerful generative capabilities of the diffusion model [33,34,35], several studies [42,43,44,45] have explored its application for BFR, and several additional studies [13,44,46,47] have also explored its application for BFR. DR2 [46] proposes a two-stage framework DR2E for blind face restoration that uses a pretrained diffusion model to remove various types of degradation and a module for detail enhancement and upsampling, and, furthermore, eliminates the need for synthetically degraded data during training. IPC [47] proposes a conditional diffusion-based BFR framework like SR3 to restore severely degraded face images. This framework employs a region-adaptive strategy that enhances restoration quality while preserving identity information. DifFace [44] establishes a posterior distribution for mapping LQ images to HQ counterparts via a pre-trained diffusion model. To achieve this, the approach estimates a transition distribution from the LQ input image to an intermediate noisy image using a diffuse estimator within the diffusion model to enhance robustness to severe degradations. Additionally, it incorporates a Markov chain that transitions the intermediate image to the HQ target image by repeatedly applying a pre-trained diffusion model, which further improves face restoration performance. Lu et al. [48] propose a diffusion-based architecture that incorporates 3D facial priors. These priors are derived from a reconstructed 3D face from an initially restored image and are integrated into the diffusion reverse process to provide structural and identity information. However, these studies have not been developed to effectively leverage reference images for further enhancement. Meanwhile, PGDiff [13] proposed a partial guidance approach that is extensible to utilizing a reference image. By incorporating identity loss into the diffusion-based restoration method, it outperforms existing diffusion-prior-based methods. Inspired by this, we also propose a method that incorporates both diffusion prior and reference prior. However, unlike PGDiff [13], which does not consider the conflicts between gradients arising from multiple losses, our approach effectively addresses these conflicts through the integration of the proposed PDGrad, leading to more consistent and high-quality results for RefBFR. 3. Proposed Method In this section, we provide a preliminary overview of the guided diffusion models to aid understanding of our proposed method in Section 3.1. We then detail the overall process of the proposed method in Section 3.2. Section 3.3 describes the proposed loss function, designed to fully leverage both the coarsely restored image and the reference image at each feature level. Lastly, in Section 3.4, our proposed PDGrad is explained, which is developed to mitigate the conflicting gradient problem. 3.1. Preliminary 3.1.1. Denoising Diffusion Probabilistic Models Recently, diffusion models [33,34,35] are one of the probabilistic generative models that have achieved remarkable success in the field of image generation. The diffusion model consists of a forward process and a reverse process. The forward process gradually adds Gaussian noise to an input image, while the reverse process removes the noise and reconstructs the image from the noisy state. For an unconditional diffusion model [33] with discrete steps T, there exists a transition distribution q(xt+1|xt) at each step t∈{1,2,3,···,T} with corresponding variance schedule βt:(1)q(xt|xt−1)=N(xt;1−βtxt−1,βtI), where xt−1 and xt are samples at time t−1 and t, respectively. xt is sampled using the reparameterization trick. xt can be sampled directly from x0:(2)xt=α¯tx0+1−α¯tϵ, where α¯t=∏i=1tαi, αt=1−βt and ϵ∼N(ϵ;0,I). The sampling process begins with a pure Gaussian noise xT∼N(xT;0,I) and gradually conducts the denoising step. Practically, the ideal denoising step is approximated by pθ(xt−1|xt) [15] as follows. (3)pθ(xt−1|xt)=N(μθ(xt,t),Σθ(xt,t)), where μθ(xt,t) represents the mean, which is obtained as a linear combination of xt and an estimated noise ϵθ(xt,t), while Σθ(xt,t) denotes the variance, a constant dependent on the pre-defined βt. From Equation (2), x^0|t can be directly computed from ϵθ as:(4)x^0|t=1α¯txt−1−α¯tα¯tϵθ(xt,t). ADM [15] introduces guided diffusion to control the sample generation of the diffusion model by leveraging an external classifier pϕ(c|x) that predicts the conditioning information c such as class label. By utilizing the classifier, the conditional distribution for denoising step in Equation (3) is approximated as a Gaussian distribution and formulated as:(5)pθ,ϕ(xt−1|xt,c)≈N(μθ(xt,t)+sΣθ(xt,t)g,Σθ(xt,t)), where s denotes the strength of classifier guidance for control. Here, the diffusion of the unconditional sampling distribution is guided by the gradient g towards conditional target c, which can be written as:(6)g=∇xlogpϕ(c|x)|x=μθ(xt,t). 3.1.2. Partial Guidance Recently, PGDiff [13] has introduced a training-free method and utilized classifier guidance on an unconditional diffusion model for face restoration by leveraging a pre-trained network through a technique called partial guidance. Specifically, PGDiff [13] decomposes a high-quality face image into smooth semantics and high-frequency details. The smooth semantics of the face are provided by the pre-trained face restoration model, such as CodeFormer [16]. For the high-frequency details, PGDiff relies on the diffusion prior. In addition, by leveraging a reference image and incorporating identity loss into the partial guidance, PGDiff enhances the preservation of personal identity. This identity information is guided using a pre-trained face recognition network, such as ArcFace [1]. 3.2. Overview of Our Method Figure 2 illustrates an overview of the proposed process. Let y∈RH×W×C be the given LQ image and yr∈RH×W×C be the reference HQ image. Our goal is to predict a HQ image x0∈RH×W×C by adjusting conflicting gradient of the loss within a guided diffusion framework [44]. To this end, following PGDiff [13], a coarsely restored image yc∈RH×W×C is first obtained by adopting a pre-trained face restoration model f(·), which can be written as:(7)yc=f(y). However, unlike PGDiff [13], which begins the reverse process from pure Gaussian noise, our method starts from xτ, sampled from yc to enhance initialization and decrease the number of sampling steps [49]. As in Equation (2), xτ can be defined as:(8)xτ=α¯τyc+1−α¯τϵ, where τ∈[0,T] is a hyperparameter that determines the starting reverse process. Also, α¯τ=∏i=1ταi and ϵ∼N(ϵ;0,I). Then, the reverse diffusion process is iteratively performed using the guided diffusion model [33] as follows:(9)xt−1∼N(μθ(xt,t)−sΣθ(xt,t)∇x^0|tLtotal,Σθ(xt,t)), where s represents the guidance strength and Σθ(xt,t) is the time-dependent constant, as defined in Equation (5). x^0|t is the predicted image at timestep t (Equation (4)). ∇x^0|tLtotal denotes the gradient of the total loss with respect to x^0|t. Details of Ltotal and the computation of the corresponding gradient ∇x^0|tLtotal are discussed in the following. 3.3. Loss Function One key goal of our method is to decompose the gradients from both the coarsely restored image and the reference image into their respective low-level and high-level components and then use these as guidance. This enables our approach to effectively leverage both low-level and high-level features from the coarsely restored image and the reference image. Our total loss Ltotal in Equation (9) at arbitrary diffusion time t is formulated by:(10)Ltotal=Llow+Lhigh, where Llow and Lhigh represent losses for low-level and high-level features, respectively. For ease of notation, we omit the denoising timestep t. The former focuses on preserving low-level information such as face shape, edges and color, while the latter focuses on promoting high-level information such as face identity. The proposed loss is computed using the coarsely restored image yc, the reference image yr, and the predicted image x^0|t (Equation (4)) at an arbitrary diffusion time t. To explicitly incorporate face information from yr and yc into the diffusion process, various levels of intermediate features are extracted from pre-trained ArcFace [1] and VGG16 [50] networks. As discussed in [51], it is well established that the intermediate output feature maps of the early layers of a well-trained network capture the low-level information of the input image, while the later layers capture higher-level features. Motivated by this, various levels of features are extracted from pre-trained convolutional networks such as ArcFace [1] and VGG16 [50]. Specifically, let {ui(z)}i=14 represent a set of features extracted using ArcFace [1], which is the face recognition network to determine whether two images are of the same identity. Here, ui(z) denotes the feature extracted from the i−th intermediate layer of the ArcFace network for the input image z∈RH×W×C. In the case of low-level loss, we additionally use the VGG16 [50] network trained on a dataset that reflects human perceptual similarity to better match human preferences. Let {vi(z)}i=15 represent a set of features extracted using the VGG16 [50], where vi(z) denotes the feature extracted from the i−th intermediate layer of the VGG16 [50] for the input image z∈RH×W×C. The specific layers used in u and v are explained in the implementation details in Section 4.1. Now, we explain each loss in detail in the following subsection. 3.3.1. Low-Level Loss The proposed low-level loss Llow is defined as sum of two losses:(11)Llow=Llowc+Llowr, where Llowc represents the loss that measures the similarity between the low-level features of x^0|t and yc. Similarly, Llowr represents the loss that ensures alignment of the low-level features between x^0|t and yr. Llowc and Llowr are defined by using pre-trained networks, including ArcFace [1] and VGG16 [50]. Concretely, Llowc is defined as:(12)Llowc=∑i=13darc(ui(x^0|t),ui(yc))+∑i=15dvgg(vi(x^0|t),vi(yc)). Here, darc(·,·) [1] is defined as:(13)darc(j1,j2)=1−j1·j2j1j2, where j1 and j2 represent input vectors to measure distance. The distance function dvgg(·,·) [50] is defined as:(14)dvgg(z1,z2)=∑l1HlWl∑h,w||wlhw⊙(z1hwl−z2hwl)||22, where z1 and z2 are input images being compared, and z1hwl and z2hwl represent the feature values of the feature maps in the l−th layer at the spatial location (h,w) for z1 and z2, respectively. Hl and Wl denote the height and width of the feature map at the l−th layer, and wlhw is a weighting factor for the feature map difference at spatial location (h,w). Symbol ⊙ represents element-wise multiplication. Similarly, to enforce the alignment of low-level features between between x0|t and yr, Llowr is formulated as:(15)Llowr=∑i=13darc(ui(x^0|t),ui(yr))+∑i=15dvgg(vi(x^0|t),vi(yr)). 3.3.2. High-Level Loss The proposed high-level loss Lhigh is designed to measure the identity similarity between x0|t and yr as well as between x0|t and yc. Accordingly, Lhigh is comprised of two loss terms as:(16)Lhigh=Lhighc+Lhighr, where Lhighc and Lhighr are defined as follows. (17)Lhighc=darc(u4(x^0|t),u4(yc)). (18)Lhighr=darc(u4(x^0|t),u4(yr)).Here, darc(·,·) refers to the cosine distance metric defined in Equation (13). 3.4. The Proposed PDGrad Inspired by the PCGrad [17], we propose a gradient adjustment method for guiding diffusion models in RefBFR. Similar to PGDiff [13], the unconditioned diffusion model is guided using classifier guidance. In this context, the gradient of each loss in Equation (10) acts as a specific guidance, defined by the following equation:(19)gtotal=glow+ghigh, where gtotal, glow, and ghigh denote the gradients ∇x^0|tLtotal, ∇x^0|tLlow, and ∇x^0|tLhigh, respectively. Unlike PCGrad [17], we select a pivot gradient for each loss and adjust the other gradient by projecting it to the normal plane of the pivot gradient when a conflicting gradient occurs. This prevents the interfering component from being applied in the pivot direction. From Equation (11), glow consists of glowc=∇x^0|tLlowc and glowr=∇x^0|tLlowr, where the former is defined using yc, and the latter is defined using yr. When the angle between glowc and glowr is larger than 90∘, that is, the cosine similarity between them is a negative value, it indicates that two gradients conflict [17]. In this case, the resultant gradient glow would be suboptimal as guidance for the guided diffusion. Thus, the proposed gradient glowpd, which replaces glow, is defined as follows:(20)glowpd=glowc+glowrifglowc·glowr≥0,glowc+kl·g^lowrotherwise. When two gradients glowc and glowr do not conflict, glowpd is the same as the original glow. When they conflict, we hypothesize that yc contains more reliable information for low-level features. Thus, as shown in Figure 3, we set the pivot direction as glowc and define g^lowr by projecting glowr onto the normal plane of glowc, which is formulated as: (21)g^lowr=glowr−glowr·glowc∥glowc∥2glowc. The weighting factor kl in Equation (20) is defined by (22)kl=1if∥glowc∥≥∥g^lowr∥,∥glowc∥∥g^lowr∥otherwise. Note that when the norm of the projected gradient g^lowr is larger than that of glowc, we clip the norm of g^lowr by controlling the value of kl. This weighting factor helps our model to focus on the low-level features of yc. Similarly, ghigh consists of ghighc=∇x^0|tLhighc and ghighr=∇x^0|tLhighr. For the high-level features, yr contains more suitable information than that of yc. In this case, we define the modified gradient ghighpd which replaces ghigh as:(23)ghighpd=ghighc+ghighrifghighc·ghighr≥0,kh·g^highc+ghighrotherwise. As shown in Figure 3, we set the pivot direction for ghigh as ghighr and define g^highc by projecting ghighc onto the normal plane of ghighr, which is formulated as:(24)g^highc=ghighc−ghighc·ghighr∥ghighr∥2ghighr. The weighting factor kh in Equation (23) is defined by (25)kh=1if∥ghighr∥≥∥g^highc∥,∥ghighr∥∥g^highc∥otherwise. When the norm of the projected gradient g^highc is larger than that of ghighr, we clip the norm of g^highc by controlling the value of kh. Finally, the total gradient gtotalpd for guiding the diffusion model can be obtained by summing glowpd in Equation (20) and ghighpd in Equation (23). It is formulated as follows:(26)gtotalpd=glowpd+ghighpd. The overall pipeline of the proposed PDGrad is described in Algorithm 1. Algorithm 1 Restoration process of PDGrad1:Input: a low-quality image y, reference image yr, a diffusion model (μθ(xt,t),Σθ(xt,t)), face restorer f(·), gradient scale s and the initial timestep τ2:Output: restored image x03:yc←f(y)4:Sample xτ from q(xτ|yc) according to Equation (8)5:for t do from τ to 16: μ,Σ←μθ(xt,t),Σθ(xt,t)7: x^0|t←1α¯txt−1−α¯tα¯tϵθ(xt,t)8: Compute gradients glowpd according to Equation (20)9: Compute gradients ghighpd according to Equation (23)10: gtotalpd←glowpd+ghighpd11: xt−1←samplefromNμ−sΣgtotalpd,Σ12:end for 1:Input: a low-quality image y, reference image yr, a diffusion model (μθ(xt,t),Σθ(xt,t)), face restorer f(·), gradient scale s and the initial timestep τ Input: a low-quality image y, reference image yr, a diffusion model (μθ(xt,t),Σθ(xt,t)), face restorer f(·), gradient scale s and the initial timestep τ 2:Output: restored image x0 Output: restored image x0 3:yc←f(y) yc←f(y) 4:Sample xτ from q(xτ|yc) according to Equation (8) Sample xτ from q(xτ|yc) according to Equation (8) 5:for t do from τ to 1 for t do from τ to 1 6: μ,Σ←μθ(xt,t),Σθ(xt,t) μ,Σ←μθ(xt,t),Σθ(xt,t) 7: x^0|t←1α¯txt−1−α¯tα¯tϵθ(xt,t) x^0|t←1α¯txt−1−α¯tα¯tϵθ(xt,t) 8: Compute gradients glowpd according to Equation (20) Compute gradients glowpd according to Equation (20) 9: Compute gradients ghighpd according to Equation (23) Compute gradients ghighpd according to Equation (23) 10: gtotalpd←glowpd+ghighpd gtotalpd←glowpd+ghighpd 11: xt−1←samplefromNμ−sΣgtotalpd,Σ xt−1←samplefromNμ−sΣgtotalpd,Σ 12:end for end for 4. Experiments As mentioned in Li et al. [10], the RefBFR methods generally outperform single-image BFR methods, since reference image contains rich textures and the fine detains lost in the given LQ image [52,53]. Hence, in this paper, we mainly compare our proposed method with the recent reference-based BFR methods such as ASFFNet [11], DMDNet [12], and PGDiff [13]. Additionally, we report comparison with the single-image BFR methods, including VQFR [39], CodeFormer [16], RestoreFormer++ [40] and DifFace [44]. All experiments in this paper are conducted using the official models with pre-trained weights provided by the authors. In this section, we provide the details of experimental settings in Section 4.1. In Section 4.2, we compare our proposed method with the state-of-the-art BFR methods through both qualitative and quantitative analyses. Section 4.4 presents the evaluation results of the ablation study to assess the effect of each component of our proposed approach. 4.1. Experimental Setting 4.1.1. Implementation Details Following PGDiff [13], we utilize the pre-trained diffusion model provided by Yue et al. [44] for a fair comparison. This model is an unconditional diffusion network trained on the FFHQ dataset [54] and supports an image resolution of 512×512. In our RefBFR process, we leverage CodeFormer [16] as a pre-trained face restoration model to obtain coarsely restored image from a given LQ image, which is denoted as f(·) in Equation (7). It is noteworthy that the proposed method employs off-the-shelf pre-trained networks which are readily accessible online, without the need for additional training. The proposed framework is implemented using Pytorch [55] and the inference process is executed on a single NVIDIA GeForce RTX 3090 GPU. Empirically, we set τ for initial guidance step to 700, and gradient scale s to 0.1. The intermediate features {ui(z)}i=13 are extracted from layers conv1_1, conv2_2, and conv3_2 of ArcFace [1]. u4(z) is final output feature of ArcFace [1]. The intermediate features {vi(z)}i=15 are extracted from layers conv1_2, conv2_2, conv3_3, conv4_3 and conv5_2 of VGG16 [50]. 4.1.2. Datasets To evaluate our method, we use CelebRef-HQ dataset [12], which comprises a total of 10,555 HQ face images. This dataset includes 1005 distinct identities and each individual has between 2 and 21 images. Specifically, for our evaluation, we randomly select two images from each of the 1005 identities in the dataset. Then, one image is designated as the ground-truth HQ image, while the other image serves as the reference HQ image. Following the degradation model specified in recent BFR studies [16,27,39,56], the LQ images are synthesized as follows:(27)y=[[(xGT⊗kσ)↓r+nδ]JPEG]↑r, where the ground-truth HQ image xGT is first blurred using a Gaussian kernel kσ, then downsampled by a scale factor r. Next, Gaussian noise nδ is added, followed by JPEG compression with quality factor q is applied. Lastly, the LQ image y is resized back to 512×512. In this paper, we randomly sample σ, r, δ, and q from [0.1,15],[24,40],[0,20], and [30,100], respectively. 4.1.3. Evaluation Metrics For a quantitative evaluation, we employ PSNR, SSIM [57] and NIQE [58], which are commonly used metrics in the image restoration field. Additionally, we measure LPIPS [50] to assess perceptual similarity between ground-truth images and restored images. Furthermore, FID [59] is used to quantify the distance between the feature distributions of HQ face datasets and restored images. We employ the CelebRef-HQ dataset [12] to measure the feature distributions of HQ face dataset. To measure the similarity in facial identity between the ground-truth images and the restored images, we compute the angle between their embedding vectors using ArcFace [1], denoted as Deg [39]. We also compare the landmark distance (LMD) [39], which is calculated as the average L2 distance of 98 facial landmarks predicted using Awing [60] between the ground-truth images and the restored images. 4.2. Quantitative Comparison The quantitative comparisons of various RefBFR methods are shown in Table 1. Here, ASSFNet [11] and DMDNet [12] are face restoration methods that incorporate landmark estimation procedures. However, for certain LQ input images, these methods struggle with accurate landmark detection, preventing them from generating results and rendering testing infeasible. Therefore, to ensure a fair comparison between methods, we conduct experiments on 662 LQ input images, a subset of the CelebRef-HQ dataset [12], where testing with ASSFNet [11] and DMDNet [12] is feasible, as shown in Table 1. The results demonstrate that the proposed PDGrad achieves better performance than other methods in terms of LPIPS, Deg, LMD and NIQE. PDGrad achieved a |0.4508−0.4437|/0.4508=1.57% better result in terms of LPIPS compared to the second best competitive method, indicating that it consistently restores faces with perceptual quality closest to the ground truth. For fidelity, the proposed PDGrad achieved the highest performance in terms of Deg and LMD, compared to the second best method with improvements of |55.53−53.1|/55.53=4.38% and |6.25−6.01|/6.25=3.84%, respectively. This demonstrates that our method can accurately recover facial identity similarity and details. Additionally, in terms of image quality metrics such as NIQE, the proposed PDGrad outperforms DMDNet [12] by |3.85−3.38|/3.85=12.21% and produces more realistic details. This can be attributed to the incorporation of (1) the proposed loss function that considers perceptual quality and identity preservation and (2) the proposed gradient adjustment procedure that effectively handles the conflict of the gradients. In Table 2, we further compared the proposed PDGrad with PGDiff [13] using the full CelebRef-HQ dataset [12], which consists of 1005 LQ input images. Compared to PGDiff, PDGrad shows outperforming results on the LPIPS, Deg, LMD and NIQE metrics. Notably, we achieved improvements of |56.68−53.90|/56.68=4.9% and |4.32−3.39|/4.32=21.53% in Deg and NIQE, respectively. This indicates that PDGrad generates face images that are more faithful to the ground truth identity while maintaining high image quality during the guided diffusion process. Table 3 provides a quantitative comparison between the proposed PDGrad and single-image BFR methods on the full CelebRef-HQ dataset [12]. The results demonstrate that our method achieves better or at least comparable performance in terms of LPIPS, Deg, LMD, NIQE and FID. While CodeFormer [16] achieves a higher perceptual quality than other methods according to LPIPS, its identity similarity is significantly compromised according to Deg. Although our proposed PDGrad is slightly worse in LPIPS compared to CodeFormer [16], it excels at preserving identity similarity, as measured by Deg. Notably, our method demonstrates a significant improvement |68.3−53.9|/68.3=21.08% in Deg compared to the second-best model. 4.3. Qualitative Comparison Visual comparisons of RefBFR and single-image BFR methods are presented in Figure 4 and Figure 5, respectively. In each figure, the even-numbered rows provide close-up views that highlight specific details in the same areas indicated by red rectangles in the LQ input of the corresponding images in the odd-numbered rows. Figure 4 demonstrates that ASFFNet [11] and DMDNet [12] fail to preserve the identity and to produce a proper facial shape. Specifically, it can be observed that components like the eyes have been mostly restored, but there are difficulties in restoring most components such as the nose and mouth. PGDiff [13] is able to produce high-quality images, but it showed a lack of facial details in terms of preserving identity. Unlike other methods, our PDGrad is able to generate high-quality images with high fidelity in skin texture, wrinkles and eye shape. In Figure 5, VQFR [39] and RestoreFormer++ [40] fail to produce satisfactory restoration results due to severe degradations. The results contain artifacts and lack facial details. CodeFormer [16], DifFace [44] and PMRF [41] produce high-quality images, but they also lack of facial details important for preserving identity. However, the proposed PDGrad exhibits superior performance over all other methods in restoring sharp and fine details of the face (e.g., in the eyes, nose and mouth). Moreover, the proposed method can generate identity-preserving results consistent with the GT while also improving the perceptual quality of the image. 4.4. Ablation Study We conducted ablation studies to investigate the impact of each component in the proposed PDGrad. First, Table 4 presents the effects of the gradient adjustment components in the PDGrad, summarizing the configurations and results for each experiment. All the methods in Table 4 use the same network architectures and a loss function defined as the sum of Equations (11) and (16), with the only difference being the gradient used to guide the diffusion model. A1 in Table 4 represents a baseline model, where the total gradient in Equation (19) is obtained by simply summing of multiple gradients without any gradient adjustment. The model is then guided by this gradient during the diffusion sampling process. A2 is a method that resolves gradient conflicts by adjusting only the gradient direction. This is achieved by projecting the gradient to the normal plane of the pivot gradient when conflicts occur. In A2, the values of both kl and kh are fixed as 1 in Equations (20) and (23) for all cases, respectively. Compared to A1, A2 shows the improvements of |0.4513−0.4499|/0.4513=0.31% in LPIPS, |59.24−53.90|/59.24=9.01% in Deg, |6.49−6.36|/6.49=2% in LMD and |3.44−3.39|/3.44=1.45% in NIQE, respectively. These improvements highlight that adjustment of the gradient direction in PDGrad effectively mitigates the conflicts between gradients arising from multiple losses. Consequently, the diffusion process is more efficiently guided, enhancing the quality of the generated images. PDGrad is our proposed method, which is built upon A2 by additionally applying an adaptive scaling for kh and kl to ensure that the magnitude of the projected gradient does not exceed that of the pivot gradient. This adaptive scaling is designed to enhance the influence of the pivot gradient when applying gradient adjustment. As a result, PDGrad not only adjusts the gradient direction towards the pivot, but also preserves the influence of the pivot gradient by adjusting the magnitude of other gradients. This leads to the restoration of images with both perceptually improved and enhanced fidelity. Consequently, compared to A2, PDGrad shows further improvements of |0.4506−0.4499|/0.4506=0.16% in LPIPS and |54.02−53.90|/54.02=0.22% in Deg. To effectively guide detailed facial information, PDGrad defines the low-level loss by combining two components, darc and dvgg, as shown in Equations (12) and (15). As shown in Table 5, to evaluate the impact of each component in these equations, we performed an additional ablation study by using either darc or dvgg in Equations (12) and (15). The experiment was performed by varying only the low-level loss component, while applying gradient adjustment, including gradient projection and adaptive scaling. A3 shows results using darc exclusively in both Equations (12) and (15), while A4 demonstrates results using dvgg exclusively in those equations. When A3 and A4 are compared to PDGrad, the results of PDGrad show an improvement in the LPIPS by |0.4616−0.4499|/0.4616=2.53% and |0.4632−0.4499|/0.4632=2.87%, respectively. Additionally, the Deg score improves by |60.60−53.90|/60.60=11.06% and |61.87−53.90|/61.87=12.88%, respectively. These improvements indicate that using both darc and dvgg together is more effective in guiding perceptual and fidelity information than using either component alone. As shown in Table 6, we conducted further experiments to explore the effects of the input images by using either the coarsely restored image yc from CodeFormer [16] or the reference image yr individually for gradient computation in the diffusion sampling process. This setup represents an extreme scenario where the angle between two input gradients derived from yc and yr, respectively, are aligned in the same direction. This alignment occurs when yc and yr are identical, resulting in the same result as if either image were used alone. In Table 6, A5 represents the case of using only yc, where glowpd and ghighpd in Equations (20) and (23) are set to glowc and ghighc, respectively. Similarly, A6 represents the case of using only yr, where glowpd and ghighpd in Equations (20) and (23) are set to glowr and ghighr, respectively. Our results confirm that PDGrad, which utilizes both yc and yr, as presented in Table 6, outperformed the other models across most metrics. 5. Conclusions In this paper, we present a Pivot Direction Gradient (PDGrad), a novel gradient adjustment method designed to enhance reference-based blind face restoration within the guided diffusion framework. By focusing on the issue of conflicting gradients in multi loss-based guidance, the proposed method aligns gradients across different feature levels, ensuring both low-level and high-level facial characteristics are accurately restored. Through comprehensive experiments, we have demonstrated that the proposed method consistently outperforms existing methods, offering robust solution for reference-based blind face restoration. This advancement highlights the potential of gradient adjustment techniques for guided-diffusion models and the broader image restoration field.