ACL: A ligned C ontrastive L earning Improves BERT and Multi-exit BERT Fine-tuning Thanks: Corresponding author. For any inquiries, please contact: michaelwzhu91@gmail.com. Affiliation: DataSelect AI, Xuhui, Shanghai, China Abstract Despite its success in self-supervised learning, contrastive learning is less studied in the supervised setting. In this work, we first use a set of pilot experiments to show that in the supervised setting, the cross-entropy loss objective (CE) and the contrastive learning objective often conflict with each other, thus hindering the applications of CL in supervised settings. To resolve this problem, we introduce a novel A ligned C ontrastive L earning (ACL) framework. First, ACL-Embed regards label embeddings as extra augmented samples with different labels and employs contrastive learning to align the label embeddings with its samples’ representations. Second, to facilitate the optimization of ACL-Embed objective combined with the CE loss, we propose ACL-Grad, which will discard the ACL-Embed term if the two objectives are in conflict. To further enhance the performances of intermediate exits of multi-exit BERT, we further propose cross-layer ACL (ACL-CL), which is to ask the teacher exit to guide the optimization of student shallow exits. Extensive experiments on the GLUE benchmark results in the following takeaways: (a) ACL-BRT outperforms or performs comparably with CE and CE+SCL on the GLUE tasks; (b) ACL, especially CL-ACL, significantly surpasses the baseline methods on the fine-tuning of multi-exit BERT, thus providing better quality-speed tradeoffs for low-latency applications. 1 1 1 Codes will be made publicly available upon acceptance. 1 Introduction Recently, contrastive learning has been proved repeatedly to be effective in unsupervised learning to obtain generic representations for downstream supervised tasks 53 ; 43 ; 42 ; 15 ; 1 ; 2 ; 48 ; 51 ; 67 ; 70 ; 60 ; 5 ; 57 ; 32 ; 89 ; 78 ; 77 ; 76 ; 23 ; 84 ; 63 ; 86 ; 13 ; 79 ; 71 ; 35 ; 65 ; 66 ; 50 ; 80 ; 91 ; 69 ; 47 ; 25 ; 22 ; 64 ; 62 ; 95 ; 94 , by asking the model to represent different views of a sample as close as possible and represent different samples as distinct as possible. Despite the majority of literature on contrastive learning works in the unsupervised learning setting, supervised contrastive learning has also been adapted to the supervised settings 21 . 21 construct the supervised contrastive learning (SCL) objective to ask the representations of examples in the same class to be close and those from different classes to be distant. The idea of SCL has also been developed in the field of NLP 12 ; 41 ; 16 ; 41 ; 16 ; 83 ; 82 ; 26 ; 54 ; 2 ; 72 ; 85 ; 10 ; 96 ; 68 ; 37 ; 88 ; 93 ; 24 ; 90 ; 81 ; 74 ; 64 ; 49 ; 29 ; 61 ; 40 . However, despite the good results, we argue that the current SCL approach still has room for improvements. One intuitive reason is that the SCL objective may not align with the CE objective, thus resulting in poor optimization. Figure 1: Our proposed ACL framework. ACL differs from the vanilla SCL objective in that it also aligns the semantic representations of input samples with the label embeddings through the contrastive learning objective. Intuitively, ACL considers the label embeddings as anchors to encourage more apparent separations of samples of different labels. Although we show a binary classification case for simplicity, the loss is generally applicable to any multi-class classification setting. In this work, we first conduct a pilot experiment to show that: the gradients of SCL and CE have quite different directions, even after being jointly optimized (see Figure 2 ). To overcome this issue, we propose a novel contrastive learning framework, A ligned C ontrastive L earning (ACL) (depicted in Figure 1 ). Our ACL framework consists of two parts: (a) ACL-Embed, which includes the learnable label embeddings from the classifier in the contrastive learning objective. ACL-Embed uses label embeddings as anchors to separate the representations from different classes. (b) ACL-Grad, which discards the ACL-Embed loss term if its gradient has a quite different direction (or angle > 90 ∘ ) than the one from CE loss. ACL-Grad makes the loss function adjustments adaptively on a given batch of samples. We further investigate whether our ACL framework can help to improve the multi-exit BERT fine-tuning. Multi-exit BERT installs an intermediate classifier (or exit) on each layer of BERT. Multi-exit BERT is the backbone for early exiting, a vital model inference speedup method 39 ; 20 ; 73 ; 92 ; 87 . Early exiting can adaptively decide how many layers of BERT the sample should go through before making the final prediction using one of the intermediate exits. Due to its ability to flexibly adjust the average latency, early exiting has attracted attention from academia and industry. To help improve the overall performances of multi-exit BERT, we propose a cross-layer ACL objective (ACL-CL), that is, to treat the last layer as the teacher and incorporate the teacher’s sample representations and label embeddings in the intermediate exit’s ACL objective. Intuitively, we use the ACL objective as a knowledge distillation approach so that the stronger teacher exit can guide the optimization direction of the weaker student exit. We conduct extensive experiments and ablation studies on the GLUE benchmark 45 with BERT 4 and RoBERTa 28 backbones. For BERT/RoBERTa fine-tuning, we show that our ACL objective achieves performance improvements over vanilla fine-tuning or CE+SCL approach 12 on 5 of the seven classification tasks. Our ACL framework is quite effective in multi-exit BERT/RoBERTa fine-tuning. With the RoBERTa backbone, our ACL-CL objective can outperform TinyBERT/DistillBERT at the 6-th layer without further pre-training. Our contributions are three-fold: • We propose ACL, a novel contrastive learning framework in supervised settings. ACL consists of two components, ACL-Embed and ACL-Grad. • We propose the cross-layer ACL objective for multi-exit BERT fine-tuning, an effective way to distill knowledge from the deeper exits. • Experiments on the GLUE benchmark demonstrate the effectiveness of our ACL framework in BERT fine-tuning and multi-exit BERT fine-tuning. 2 Related Work 2.1 Contrastive learning Contrastive learning in the supervised setting 2 2 2 See Appendix A for related works on contrastive learning in the unsupervised settings Contrastive learning has been extended to the supervised setting. 21 propose the supervised contrastive loss (SCL) that pulls the representations of images from the same class closer while pushing those from different classes further. 12 also develops a version of SCL in the context of BERT fine-tuning. Our work complements this branch of literature by extending 12 to propose a novel supervised contrastive learning objective, ACL, which includes the label embedding from the classifiers in the contrastive objectives. In addition, we propose stabilizing the CL and CE objective joint optimization adaptively based on gradient directions. 2.2 Multi-Exit BERT training Adaptive inference method like early exiting 39 ; 20 ; 73 ; 92 ; 87 is an important branch of literature focusing on applying different parts of BERT to process different samples, achieving inference speedup on average. It is in parallel with and can work together with static model compression methods 38 like network pruning 75 ; 58 ; 6 ; 11 , student network distillation 36 ; 33 ; 18 . Early exiting requires a multi-exit BERT, a BERT backbone with an intermediate classifier (or exit) installed on each layer. Early exiting literature mainly focuses on the design of the early exiting strategies, that is, determining when an intermediate exit’s prediction is suitable as the final model prediction. Score based strategies 39 ; 55 ; 20 ; 56 , prior based strategies 37 and patience based strategies 73 have been proposed. However, the literature focuses less on the training of the multi-exit neural network, which is a major factor for early exiting performances. There are three types of training methods for training the multi-exit neural network: (a) joint training (JT) 39 , that is, all the exits are jointed optimized together with the fine-tuning of BERT. (b) two-stage training method (2ST) 27 ; 55 , which first fine-tune the backbone BERT and the last layer’s exit till convergence, and then at the second stage, the backbone and the last exit are fixed, and the intermediate exit is trained by distilling knowledge from the last exit. (c) Berxit 56 propose an alternating training (ALT) method, combining 2ST and JT. Our work complements the literature on Early exiting or multi-exit neural networks by applying cross-layer contrastive learning (our ACL-CL approach) to exploit the potentials of shallow BERT layers better and improve the overall performances of multi-exit BERT. 3 Preliminaries In this section, we introduce the necessary background for BERT early exiting. we consider the case of multi-class classification with K classes, \mathcal{K}=\left\{1,2,...,K\right\} . During training, the model will receive a batch \left\{(x_{i},y_{i}),i\in\mathcal{I}=\left\{1,2,...,N\right\}\right\} containing N samples, where x_{i}\in\mathbf{R}^{L} is the input sentence consisting of L words, and y_{i}\in\mathcal{K} is the label. 3.1 Contrastive learning in the supervised setting 21 and 12 incorporates samples’ label information to CL in a straightforward way. 3 3 3 Reader can refer to Appendix B.1 for a brief introduction to the contrastive learning objective for the unsupervised setting. It simply takes the inputs with the same label in the batch as positive samples and those from different classes as negative samples. Let \Phi(\cdot)\in\mathbf{R}^{d} denotes an encoder that outputs the l_{2} normalized final representation; N_{y_{i}} is the total number of examples in the batch that have the same label as y_{i} ; \tau is the scalar temperature parameter that controls the separation of classes. The supervised contrastive objective is: \displaystyle s(i,j) \displaystyle=\dfrac{\exp(\Phi(x_{i})\cdot\Phi(x_{j})/\tau)}{\sum_{k=1}^{N}\exp(\Phi(x_{i})\cdot\Phi(x_{k})/\tau)}, \displaystyle\mathcal{L}_{sup} \displaystyle=\sum_{i=1}^{N}\dfrac{-1}{N_{y_{i}}-1}\sum_{j=1}^{N}\mathbf{1}_{i\neq j}\mathbf{1}_{y_{i}=y_{j}}\log s(i,j). Let \lambda denote a scalar weighting hyperparameter, then the overall loss is a weighed average of the CE loss (denoted as \mathcal{L}_{ce} ) and the SCL loss: \mathcal{L}=(1-\lambda)\mathcal{L}_{ce}+\lambda\mathcal{L}_{sup} . Supervised contrastive learning motivates to learn the generic semantic representations that help separate the samples into different classes. However, we are asking whether the optimization of CL and CE loss terms will always agree with one another? In this work, we will look closely into this aspect of SCL. 3.2 Early Exiting Early exiting is based on multi-exit BERT, which are BERT with classifiers (or exits) at each layer. With M layers, M classifiers f_{m}(x;\theta_{m}):\mathcal{X}\rightarrow\Delta^{K}(m=1,2,...M) are designated at M layers of BERT, each of which maps its input to the probability simplex \Delta^{K} . f_{m}(x;\theta_{m}) can take the form of a simple linear layer (linear exits) following 73 . However, in this work, we find that adding a multi-head self-attention layer to the exit (MHA exit) can boost the performances of intermediate layers. Representations of the current BERT layer, H^{(m)}_{i} , will first be down-sampled to a smaller dimension d_{e} (e.g., 64). Then, it will go through a Tanh activation, followed by a multi-head self-attention operation 44 . Then the [CLS] token’s vector representation will be passed to the second Tanh activation, and the final representation in the exit is denoted as h^{e}_{i} . h^{e}_{i} will be fed into a linear layer to generate the predicted logits. This work considers the MHA exit as the default exit architecture. 3.3 Multi-exit BERT training Due to the length limit, readers are referred to Appendix B.2 for other multi-exit BERT training methods. Now, we introduce the 2ST training method. 2ST . The two-stage (2ST) 55 ; 27 training strategy 4 4 4 Due to length limit, reader are referred to Appendix B.2 for other multi-exit BERT training methods. divides the training procedure into two stages. The first stage is identical to the vanilla BERT fine-tuning, updating the backbone model and only the final exit. In the second stage, we freeze all parameters updated in the first stage and fine-tune the remaining exits separately: \displaystyle\mathrm{Stage1}:\mathcal{L}_{stage1}=\mathcal{L}^{CE}_{M}(y_{i},f_{M}(x_{i};\theta_{M})) \displaystyle\mathrm{Stage2}:\mathcal{L}_{stage2}=\mathcal{L}_{m}^{CE},m=1,...,M-1. where \mathcal{L}^{CE}_{m}=\mathcal{L}_{m}^{CE}(y_{i},f_{m}(x_{i};\theta_{m})) denotes the cross-entropy loss of m -th exit. The 2ST training’s performances can be further enhanced by treating the last exit trained in the first stage as the teacher model and conducting knowledge ditillation 17 , as is demonstrated by 27 . 4 Aligned Contrastive Learning In this section, we elaborate on our ACL framework. The core idea of ACL is to make the contrastive learning objective in agreement with the learning of the classifier. 4.1 Motivation Combining an SCL objective with the CE loss could help the BERT backbone model extract generic semantic representations. However, through our preliminary experiments, we observe that the weight of the SCL loss \lambda has a significant impact on the model performances, and the better performances are achieved by setting \lambda to be relatively small (e.g., 0.02, 0.01). Intuitively, it seems that the SCL objective conflicts with CE loss to a certain degree. To visualize the relations and interactions between CE loss and SCL loss, we conduct a pilot experiment on the RTE task from GLUE benchmark 46 . We fine-tune two BERT models on RTE. Model A employs only the CE objective for fine-tuning, while model B uses the CE+SCL objective, following the settings from 12 . We run the fine-tuning procedures five times under different random seeds. Model A achieves the accuracy of 65.3 on average (with std. 0.35), and model B achieves the 65.8 on average (with std. 0.84). After adding the SCL objective, model B’s performance becomes more unstable than vanilla BERT fine-tuning. To take a closer look into the interactions between CE loss and SCL loss, we separately compute the gradients derived from the two loss objective, on the fine-tuned model A (or B) with model parameters fixed, \mathbf{g}_{SCL}=\Delta_{\Theta}\mathcal{L}_{SCL} and \mathbf{g}_{CE}=\Delta_{\Theta}\mathcal{L}_{CE} . Their angle is given by \displaystyle\cos\gamma \displaystyle=\dfrac{\mathbf{g}_{SCL}\cdot\mathbf{g}_{CE}}{|\mathbf{g}_{SCL}|\cdot|\mathbf{g}_{CE}|}, \displaystyle\gamma \displaystyle=Argcos(\cos\gamma). (2) The distribution of the gradient angles are plotted in Figure 2 . We can see that, for a BERT fine-tuned solely with CE loss, the gradients’ angle \gamma ranges from 75 ∘ to 105 ∘ , meaning that they have conflicting directions for optimization. The same observation can be made for BERT fined-tuned with the CE+SCL objective. Joint optimization does not make the SCL objective more in alignment with the CE loss in terms of optimization directions. In summary, the above observation motivates us to align the two objectives’ optimization directions, and as a result, we may obtain better convergence. Figure 2: The distribution of gradient angles after the BERT model is fine-tuned on RTE with the CE objective (in blue) or the CE+SCL objective (in orange). 4.2 ACL-Embed Now we introduce our first modification to the SCL objective, which is called a ligned c ontrastive l earning via label embed ding (ACL-Embed). Label embedding The sentence representation h^{e}_{i} given out by the MHA exit will be fed into the classification layer to calculate the logits \mathrm{logits}_{i}=h^{e}_{i}W_{c}, (3) where W\in\mathbf{R}^{d_{e}\times K} is the learn-able parameter matrix of the classifier. Denote the k -th column vector of W as l_{k}\in\mathbf{R}^{d_{e}} . The CE loss objective is to maximize the dot product between h^{e}_{i} and l_{y_{i}} while minimizing the dot products between h^{e}_{i} and the other column vectors of W . Thus, one can treat l_{k} as a vector representation of label k . Aligned contrastive loss Note that SCL 12 calculates the contrastive learning objective among the samples’ representations. With the MHA exit, we consider h^{e}_{i} as the sample representation and feed it into the SCL objective. In contrast, we consider l_{k} ( k\in\mathcal{K} ) as additional K samples \left\{(l_{k},k),k\in\mathcal{K}\right\} to augment the current batch of samples. Thus, we will calculate the new contrastive loss \mathcal{L}_{acl} on the augmented batch of N+K samples via Equation . We will refer to this objective as the ACL objective. And the overall loss is given by \mathcal{L}=(1-\lambda)\mathcal{L}_{ce}+\lambda\mathcal{L}_{acl} . The idea of ACL-Embed is quite intuitive (depicted in Figure 1 ). First, the SCL objective is separated from CE loss. Meanwhile, our ACL-Embed objective combines the learning of the classifier with the contrastive loss. In this way, label embeddings serve as anchors to encourage the representations to cluster around them and thus separate away from the other classes. Note that ACL-Embed consists of three parts: (a) contrast among samples, which learns the generic representation across samples; (b) contrast between sample representation and the label embeddings learn to align the representations with the classification task; (c) contrast among the label embeddings, which serves as regularization for the classifier parameters. 4.3 ACL-Grad Although our ACL-Embed method can align the CE objective and the CL-based objective to a certain extent, we still find that, during fine-tuning, the angles between \mathbf{g}_{ACL} and \mathbf{g}_{CE} may be more than 90 ∘ in around 20% of the mini-batches. Thus, we propose a modification to the optimization procedure, called a ligned c ontrastive l earning via g radients (ACL-Grad). ACL-Grad adaptively modifies the co-efficient \lambda according to the gradient angle. Denote a threshold \gamma_{thres} for gradient angles, then the weight of \mathcal{L}_{acl} on the current batch is given by: \lambda^{{}^{\prime}}=\left\{\begin{array}[]{lr}\lambda,&\text{if gradient angle}\ \ \gamma\leq\gamma_{thres}\\ 0,&\text{otherwise}.\end{array}\right. (4) Intuitively, the gradient descent direction mainly follows CE loss. ACL-Embed provides an auxiliary regularization to the optimization. On a batch of samples, if the gradient angle is larger than \gamma_{thres} , we will discard the gradient information from ACL-Embed to avoid being distracted. 4.4 Cross-layer ACL as cross-sample knowledge distillation This work mainly adopts the 2ST training strategy for multi-exit BERT fine-tuning (in Section 3.3 ). After the BERT backbone and the last layer’s exit is fine-tuned, the intermediate exits will distill knowledge from the last exit to gain better performances. However, we believe that the usual knowledge distillation method like 17 , feature distillation, or attention matrix distillation 18 does not provide knowledge of the semantic relations or comparisons among different samples. In addition, the intermediate classifiers do not receive direct guidance from the last exits. The above analysis encourages us to propose a novel knowledge distillation method to discover the potentials of intermediate layers better. Assume we are now fine-tuning the m -th layer’s exit of BERT, which represent the batch of samples into vectors h_{i}^{e,m} ( i\in\mathcal{I} ). Its classification layer has label embedding l_{k}^{m} . Despite the parameters are fixed, the last exit M can provide representations h_{i}^{e,M} ( i\in\mathcal{I} ) for the batch of samples and the label embeddings l_{k}^{M} ( k\in\mathcal{K} ). We propose a novel cross-layer contrastive learning objective, ACL-CL. ACL-CL calculates the contrastive loss using Equation on the augmented batch of 2N+2K representations, including h_{i}^{e,m} , l_{k}^{m} , h_{i}^{e,M} and l_{k}^{M} . During the training of an intermediate exit, ACL-CL can also incorporate ACL-Grad method to make the optimization smoother. Intuitively, ACL-CL can be beneficial due to following three reasons. • First, ACL-CL objective includes the ACL-Embed objective on the intermediate layer. • Second, the shallow exit can directly learn from the label embeddings from the last exit, thus improving its representation quality. • Third, the shallow layer of BERT is less capable of learning a good representation. However, with our ACL-CL objective, it can learn generic features among samples from the deeper layer. ACL-CL can be seen as an effective knowledge distillation method for the intermediate layers, which does not only extract knowledge sample by sample but also distill knowledge on how the samples are contrasted and how to form better label embeddings for classification. 5 Experiments 5.1 Datasets We evaluate our proposed approach to the classification tasks on GLUE benchmark 46 . We only exclude the STS-B task since it is a regression task, and we exclude the WNLI task following previous work ( 3 ; 19 ; 59 ) . 5.2 Baseline methods In the normal BERT fine-tuning scenario, we compare ourselves with two strong baselines: (1) fine-tuning BERT/RoBERTa with CE loss 3 ; (2) fine-tuning with CE+SCL objective 12 . In order to demonstrate our ACL’s effectiveness on multi-exit BERT training and inference speedup, we compare with the following types of baselines: Directly training of intermediate layers We experiment with the naive baseline that directly utilizes the first 6 layers of the original BERT/RoBERTa with an MHA exit on the top, denoted by BERT-6L/RoBERTa-6L respectively. Model compression methods For model parameter pruning, we include the results of LayerDrop ( 7 ) and attention head pruning ( 31 ) . For knowledge distillation based pre-training, we include DistillBERT ( 34 ) and TinyBERT 19 . For module replacing, we include BERT-of-Theseus ( 59 ) . Existing training methods for multi-exit BERT We compare our ACL framework with three representative multi-exit BERT training methods: (a) JT 73 ; 39 ; (b) 2ST 55 ; 27 ; (c) ALT 56 . We implement 2ST with both the linear exit and the MHA exit. 5.3 Experimental settings Devices We implement our ACL framework on the base of HuggingFace’s Transformers. We conduct our experiments on Nvidia V100 16GB GPUs. Backbone models All of the experiments are built upon the Google BERT base 3 or RoBERTa base 28 models. Hyper-parameter settings The hidden dimension d_{e} in the MHA is set to 64 so that the extra parameters introduced by MHA exits will be less than 0.2% of the pre-trained models. We use the AdamW optimizer 30 to finetune the pre-trained models with a 0.01 weight decay. We train each backbone model and intermediate exits for 15 epochs and use a linear learning rate decay from 2e-5 to 0 with 10% of the total optimization steps as warm-up. We set the dropout rate to 0.1 for all layers. Batch size and gradient accumulation are used so that there are around two hundred gradient update steps per epoch. The temperature factor \tau is chosen as 0.5, the coefficient \lambda is set at 0.02, and the gradient angle threshold is set to 90 ∘ . We mainly adopt the 2ST training method with MHA exits for multi-exit BERT/RoBERTa training. Other hyper-parameters follow the Huggingface Transformers 52 . 5.4 Overall Comparison Model Method #Flops CoLA MNLI MRPC QNLI QQP RTE SST-2 Full model BERT CE 6.6G 57.3 82.9/83.4 89.6 89.5 88.7 65.3 92.8 CE+SCL 6.6G 57.9 82.6/82.7 88.7 89.7 88.5 65.8 92.1 ACL 6.6G 58.5 83.0/83.7 89.5 89.7 88.9 66.2 92.7 RoBERTa CE 6.7G 62.4 86.9/86.5 90.6 91.8 90.3 73.6 93.7 CE+SCL 6.7G 62.7 87.1/86.7 90.7 92.1 89.5 74.2 93.6 ACL 6.7G 63.1 86.8/86.9 90.8 92.3 90.2 75.3 94.2 Efficient models BERT-6L - 3.3G 43.2 81.4/81.6 84.9 87.9 87.2 61.0 90.3 RoBERTa-6L - 3.4G 45.6 82.5/83.9 86.9 89.8 88.6 67.8 91.6 LayerDrop-6L - 3.3G 43.7 82.3/82.4 86.5 89.3 89.1 67.3 91.8 HeadPrune - 4.7G 45.7 78.0/79.7 80.2 86.1 84.7 64.5 91.0 DistilBERT (6L) - 3.3G 46.1 82.1/82.0 86.2 88.3 88.5 64.1 91.3 TinyBERT (6L) - 3.3G 46.3 83.6/83.8 87.2 90.3 89.1 68.6 91.9 BERT-of-Theseus - 3.3G 44.7 81.4/81.9 85.3 88.1 88.6 66.2 91.5 ACL-BERT-6L ACL-CL 3.3G 44.7 82.0/82.1 85.1 88.5 87.7 63.6 90.8 ACL-RoBERTa-6L ACL-CL 3.3G 46.2 83.3/83.5 87.5 90.5 88.9 69.5 92.3 Table 1: Our ACL framework and baselines’ performance on the GLUE tasks. In this table, we mainly compare the performances of ACL on the full BERT/RoBERTa or the 6-th layer against baselines. Performance metrics for each task strictly follow the settings of the GLUE benchmark. Table 1 reports the main results on the GLUE benchmark with BERT and RoBERTa as the backbone models. The last two lines of the table are the 6-th layer’s performances of the multi-exit BERT/RoBERTa enhanced by our ACL framework. All baseline models are run with the original authors’ open-sourced codes. Performance comparisons for BERT fine-tuning From Table 1 , we can see that by adding our ACL objective, BERT obtains performance improvements against vanilla fine-tuning or CE+SCL on 5 of the 7 GLUE tasks. RoBERTa also benefits from the ACL objective. Performance comparisons of efficient models Table 1 also reports the comparison among model inference speedup methods. With the help of our ACL-CL objective, the intermediate exits of BERT can learn better semantic representations for the samples and the labels. As a result, ACL-BERT-6L can outperform BERT-6L. With a better backbone, ACL-RoBERTa-6L can outperform or perform comparably with the SOTA static BERT compression methods like TinyBERT. Note that TinyBERT requires that the compressed model be further pre-trained, while our method does not. 5.5 Comparisons against multi-exit BERT training methods To further demonstrate our method’s effectiveness in improving the intermediate exits, we compare our ACL-RoBERTa with the previous SOTA multi-exit BERT training methods. For visualization, we plot the curve between the layer depth and the performance score (layer-score curve) in Figure 3 . 5 5 5 Due to the length limit, layer-score curves for other GLUE tasks can be found in Figure 4 of Appendix C . The results show that our ACL-RoBERTa outperforms the baseline multi-exit BERT training methods on most of the layers. Note that the performance margins on the shallow exits are more significant than those on the deeper exits, showing that our ACL framework effectively distills knowledge from the deeper exit to guide the shallow ones. In addition, from figure 3 , we can clearly see that MHA exit significantly improve the performance of 2ST training with a simple linear head. To show that our ACL framework’s performance gains in multi-exit BERT training do not rely on the specific backbone models, we also plot the layer-score curves on the RTE and MRPC tasks with the BERT-base backbone (Figure 5 in Appendix C ). The results show that ACL-BERT can also outperform the baseline multi-exit BERT training methods. (a) MRPC (b) RTE (c) SST-2 Figure 3: The layer-score curves for different multi-exit BERT training methods, with the RoBERTa backbone. TinyBERT is represented as a dot since it is a static compression method. 5.6 Ablation on the ACL framework In order to analyze which components of our ACL framework are beneficial for multi-exit BERT fine-tuning, we step by step reduce our whole ACL framework into the vanilla 2ST training: (a) first, not use ACL objective during the backbone fine-tuning in the first stage; (b) second, reduce the ACL-CL to ACL-Embed on the intermediate exits, so the last exit will only provide logits as soft targets; (c) third, further exclude the ACL-Grad module for intermediate exits’ training; (d) fourth, substitute the ACL-Embed objective into CE+SCL; (e) fifth, drop the SCL objective. The ablation study is done on MRPC, RTE, and SST-2 tasks, with RoBERTa backbone. Table 2 reports the average performance scores across all the layers of the RoBERTa backbone. The results show that each component in our whole ACL framework is beneficial for the multi-exit BERT training procedure. Our ACL objectives can effectively guide the optimization of weak intermediate layers and result in better performances. Method cross-layer average score - MRPC RTE SST-2 ACL-RoBERTa 86.9 69.8 91.3 - CE backbone 86.9 69.7 91.1 - w/o ACL-CL 86.7 69.4 91.0 - w/o ACL-Grad 86.5 69.3 90.8 - CE+SCL 86.4 69.3 90.5 - 2ST 86.3 69.2 90.6 Table 2: Ablation study on the components of our ACL method. Cross-layer average performance scores are reported. 5.7 Visualization of the intermediate exits’ representations To demonstrate how our ACL objective improves the semantic representations, we draw the t-SNE plots of the semantic representations on the 3-th exit, with CE+SCL objective (in Figure 6(a) in Appendix D ) or with our ACL method (in Figure 6(b) in Appendix D ). The samples are from the dev set of MRPC. With CE+SCL objective, the 3-th exit achieves the performance score of 82.8%, while the score is 83.6% under the ACL method. As is shown in Figure 6(a) and 6(b) , our ACL help the model to learn more discriminative representations for the input samples, and the label embeddings learned with ACL are in alignment with the input samples from the corresponding classes. 5.8 Parameter sensitivity of the \lambda coefficient We now look into how the value of co-efficient \lambda affects the multi-exit BERT training by setting \lambda to be one of { 0.005, 0.01, 0.02, 0.1, 0.5, 1.0} for both the CE+SCL method and our ACL method. The cross-layer average scores are reported in Table 3 in the Appendix E . The results show that our ACL framework is not sensitive to the values of \lambda , whereas the CE+SCL method is prone to be affected by \lambda . This observation shows that our ACL method can conveniently apply to other tasks. 6 Conclusion In this work, we take a close look at the supervised contrastive learning objective 12 and find that the CE objective and SCL objective often conflict with each other, thus hindering better optimization. To overcome this issue, we propose the A ligned C ontrastive L earning (ACL) framework. ACL consists of three main components: (a) ACL-Embed, which aligns the label embeddings from the classifier with the samples’ representations; (b) ACL-Grad, which selectively shuts down the ACL-Embed loss term if gradients from ACL-Embed and CE have very different directions; (c) ACL-CL, which performs cross-layer contrastive learning at the shallow exit to extract knowledge from the deeper exit. Extensive experiments on the GLUE benchmark results demonstrate that: (a) our ACL framework outperforms the CE+SCL approach in BERT fine-tuning; (b) ACL-CL outperforms the existing SOTA multi-exit BERT training method. Limitation Although our ACL framework is shown to be effective in improving the multi-exit BERT training, it still has certain limitations that need to be addressed in the future: (a) In this work, we demonstrate our framework’s performances on the sentence classification tasks or sentence pair classification tasks. In future works, we would like to extend our work to broader tasks such as sequence labeling, relation extraction, and dependency parsing. (b) Our ACL framework effectively improves the training of multi-exit BERT, which suggests its value in the study of BERT inference speedup methods. We would like to investigate whether our ACL can extend to static model compression methods like the TinyBERT framework. Ethics Statement Our ACL-BERT is designated to improve the fine-tuning of BERT models and the training of multi-exit BERT, providing a better backbone model for dynamic early exiting. Our work could help make the applications of natural language processing more accessible by reducing computational costs. In addition, this work can be seen as an effort to reduce BERT-based models’ carbon footprints. Furthermore, It does not introduce new ethical concerns. Appendix A Additional related works Contrastive learning in the unsupervised setting Contrastive learning was first proposed by 14 . Since the seminal work of InfoNCE 43 , contrastive learning has been widely applied in the self-supervised learning settings 53 ; 43 ; 42 ; 15 ; 1 . MoCo 15 proposes a dictionary to maintain a negative sample set, thus increasing the number of negative sample pairs. 1 construct a positive-only contrastive learning framework, by using a stop-gradient operation to prevent collapsing solutions. Contrastive learning is also widely applied in NLP. CERT 8 use back-translations to generate positive views of an sentence and apply the MoCo framework for contrastive learning based pre-training. SimCSE 9 propose to use two separate forward passes with dropout to generate positive views and apply CL to unsupervised sentence representations. Appendix B Additional preliminaries B.1 Contrastive objective in the unsupervised setting The effectiveness of contrastive learning is widely confirmed in the self-supervised settings 42 ; 15 . Here we give a brief introduction of the self-supervised InfoNCE 43 objective widely used in the contrastive learning literature. Given N training samples \{x_{i}\}_{i=1}^{N} with a number of augmented samples, where each sample has at least one augmented sample in the dataset. Let j(i) be the index of the augmented one derived from the i -th sample, the standard contrastive loss is defined as: \mathcal{L}_{self}=\dfrac{1}{N}\sum_{i\in\mathcal{I}}-\log\dfrac{\exp(z_{i}\cdot z_{j(i)}/\tau)}{\sum_{a\in\mathcal{A}_{i}}\exp(z_{i}\cdot z_{a}/\tau)}, (5) where z_{i} is the normalized representation of x_{i} , \mathcal{A}_{i}\coloneqq\mathcal{I}\ \{i\} is the set of indexes of the contrastive samples, the \cdot operation denotes the dot product and \tau\in\mathbf{R}^{+} is the temperature factor. Here, contrastive learning treats the i -th sample as an anchor, the j(i) -th sample is a positive sample and the remaining 2N-2 samples are negative samples regarding to the i -th sample. B.2 Introduction to the multi-exit training methods Here we introduce two training methods of multi-exit BERT: JT and ALT. JT . Perhaps the most straightforward fine-tuning strategy is to minimize the sum of all classifiers’ loss functions and jointly update all parameters in the process. We refer to this strategy as JT. The loss function is: \mathcal{L}_{JT}=\sum_{m=1}^{M}\mathcal{L}_{m}^{CE} (6) where \mathcal{L}_{m}^{CE}=\mathcal{L}_{m}^{CE}(y,f_{m}(x;\theta_{m})) denotes the cross-entropy loss of the m-th exit. This method is adopted by 39 ; 20 ; 73 ; 92 ALT . It alternates between two objectives (taken from Equation 6 and 3.3 ) across different epochs, and it was proposed by BERxiT 56 : \displaystyle\mathrm{Odd}:\mathcal{L}_{stage1}=\mathcal{L}^{CE}_{M}(y_{i},f_{M}(x_{i};\theta_{M})) (7) \displaystyle\mathrm{Even}:\mathcal{L}_{joint}=\sum_{m=1}^{M}\mathcal{L}_{m}^{CE} (8) Appendix C Layer-score curves for GLUE tasks In Figure 3 , we present the layer-score curves for 3 of the GLUE tasks, to compare MEAL-BERT with other baseline models. Here, we present layer-score curves on the rest 4 tasks (CoLA, QNLI, QQP, MNLI) in Figure 4 . (a) CoLA (b) QNLI (c) QQP (d) MNLI Figure 4: The layer-score curves for different multi-exit BERT training methods, with the RoBERTa backbone. The x -axis is the depth of the exit (or the number of layers before entering this exit), the y -axis is the performance metrics following GLUE ( 45 ) . The performance of TinyBERT is represented by a dot since it is a static model compression method. In addition, we present the layer-score curves when the BERT-base is the backbone model, in Figure 5 . The results show that our ACL framework also outperform other baseline multi-exit BERT training method with BERT base as the backbone. (a) MRPC (b) RTE Figure 5: The layer-score curves for different multi-exit BERT training methods, with the BERT-base backbone. Appendix D The t-SNE plots on the third exit The t-SNE plots of the semantic representations on the 3-th exit, with CE+SCL objective (in Figure 6(a) ) or with our ACL method (in Figure 6(b) ). (a) CE+SCL (b) ACL Figure 6: The t-SNE plots of the semantic representations on the 3-th exit of RoBERTa fine-tuned on the MRPC task, with (a) the CE+SCL objective, or (b) our ACL objective. \lambda CE+SCL ACL 0.005 69.1 69.7 0.01 69.3 69.8 0.02 69.3 69.8 0.1 69.0 69.7 0.5 68.8 69.7 1.0 68.6 69.6 Table 3: The parameter sensitivity study of the co-efficient \lambda on the RTE task. Cross-layer average scores are reported. Appendix E Results for the parameter sensitivity study In Table 3 , we will report the cross-layer average performance scores, when the weight \lambda varies for the CE+SCL method, or our ACL method. The backbone model is RoBERTa, and we use the RTE task for this parameter sensitivity study.