Product-of-Experts Training Reduces Dataset Artifacts in Natural Language Inference
Aby Mathew · 2026
Read Product-of-Experts Training Reduces Dataset Artifacts in Natural Language Inference by Aby Mathew (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-19069.txt.
Introduction
Mathew presents Product-of-Experts training, which discounts examples where biased models are overconfident. A hypothesis-only model still scores 57.7 percent on SNLI, showing strong spurious correlations, and about 39 percent of baseline errors trace to artifacts. It nearly preserves accuracy at about 89 percent while cutting bias reliance, with bias agreement falling from about 50 percent to 45 percent. An ablation finds lambda of one and a half best trades debiasing against accuracy. Behavioral tests still show negation and numerical reasoning issues.
Catalog record
| CREATOR | Aby Mathew |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36694 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.19069
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Product-of-Experts Training Reduces Dataset Artifacts in Natural Language Inference Aby Mathew Abstract Neural NLI models overfit data set artifacts instead of truly reasoning. A hypothesis-only model gets 57.7% in SNLI, showing strong spurious correlations, and 38.6% of the baseline errors are the result of these artifacts. The authors propose Product-of-Experts (PoE) training, which downweights examples where biased models are overconfident. PoE nearly preserves accuracy (89.10% vs. 89.30%) while cutting bias reliance by 4.71% (bias agreement 49.85% → \rightarrow 45%). An ablation finds λ = 1.5 \lambda=1.5 that best balances debiasing and accuracy. Behavioral tests still reveal issues with negation and numerical reasoning. 1 Credits The Product-of-Experts (PoE) debiasing approach is based on the work of Clark et al. (2019). I acknowledge the HuggingFace team for their Transformers library and the creators of the SNLI dataset. All experiments were performed using personally managed computing resources, and no external funded compute was used. 2 Introduction Natural Language Inference (NLI) tasks require determining whether a premise logically supports, contradicts, or remains unrelated to a hypothesis . Modern NLI models built on pre-trained transformers such as BERT [ 5 ] and ELECTRA [ 4 ] achieve impressive performance on benchmark datasets, often surpassing traditional feature-based or shallow neural approaches. To effectively train these models, practitioners must navigate critical decisions about training objectives, model architecture, and data quality. Previously published experimental results have demonstrated that pre-trained models substantially outperform traditional approaches on NLI tasks [ 5 , 4 ] , highlighting the importance of transfer learning. However, there is a tradeoff: high benchmark accuracy does not guaranty genuine reasoning capabilities. Recent work reveals that models often exploit spurious correlations in training data patterns that enable correct predictions without semantic understanding [ 9 ] . For example, hypothesis-only models that never see premise information can achieve surprisingly high accuracy by learning annotation artifacts. This reliance on artifacts limits robustness under distribution shifts or adversarial examples where these patterns differ, raising concerns about generalization. The project demonstrates that dataset artifacts in SNLI actively harm model performance through systematic analysis. My hypothesis-only model achieves 57.7\% accuracy (compared to a 33.3\% random baseline), and we find that 38.6\% of baseline errors are artifact-driven. This model operates in three steps: 1. Take only the hypothesis text as input, completely ignoring the premise. 2. Pass the hypothesis through a pre-trained ELECTRA encoder. 3. Perform classification on the final representation. While this simple model highlights the strength of dataset artifacts, it also motivates debiasing strategies. One promising approach is Product-of-Experts (PoE) training, which applies a novel debiasing mechanism: for each training instance, examples where the bias model shows high confidence are downweighted before computing the loss. This effectively reduces the influence of spurious correlations while preserving the contribution of informative examples. We evaluate PoE on full SNLI training in Section 4. The model achieves 89.10\% accuracy compared to 89.30\% for standard training a minimal decrease ( -0.20\% ) while reducing bias reliance by 4.85 points (bias agreement: 45.0\% vs. 49.85\% ). These results demonstrate that for NLI, the choice of training objective is crucial: PoE training enables effective debiasing with negligible performance cost. Furthermore, our ablation study identifies \lambda=1.5 as the optimal debiasing strength, balancing accuracy and fairness. A qualitative analysis of behavioral tests suggests that the model works by learning robust premise-hypothesis interactions. Nevertheless, error analysis reveals that even debiased models struggle with phenomena such as negation, numerical reasoning, and compositional semantics. This indicates that while PoE reduces reliance on artifacts, it does not fully solve the deeper reasoning challenges inherent in NLI. Future work may explore hybrid architectures that combine PoE with syntactic modeling, curriculum learning, or adversarial data augmentation to further enhance robustness. In summary, the findings highlight both the promise and limitations of current NLI systems. Pre-trained transformers provide strong baselines, but without careful attention to dataset artifacts and training objectives, models risk overfitting to superficial patterns. Debiasing strategies such as PoE represent a step forward, offering improved calibration and fairness, yet continued research is needed to achieve true semantic reasoning in NLI. 3 Standard Training vs. Debiased Training The central objective is to combine the robustness of debiased models with the accuracy achieved through standard training. While traditional supervised learning pipelines for Natural Language Inference (NLI) have achieved remarkable benchmark scores, they often fail to generalize beyond the narrow distribution of training data. Debiasing methods attempt to correct this by explicitly addressing dataset artifacts and spurious correlations that standard training tends to exploit. We describe a class of artifact exploitation models dubbed hypothesis-only baselines . We then explore more sophisticated debiasing methods crafted to avoid the pitfalls aligned with standard training on biased datasets. Finally, we present the Product-of-Experts (PoE) approach, which modifies the training objective to downweight artifact-heavy examples and achieves performance on par with standard training while significantly reducing bias reliance. This comparative analysis highlights the tradeoffs between efficiency, accuracy, and robustness in modern NLI systems. 3.1 Hypothesis-Only Baseline Models To keep things simple, consider NLI classification: map an input premise-hypothesis pair to one of three labels (entailment, contradiction, neutral). A hypothesis-only baseline applies a biased composition function g to only the hypothesis sequence, completely ignoring the premise. The hypothesis tokens are encoded through a pre-trained transformer to produce a representation z would serve as an input to a classification layer. In our instantiation of the hypothesis-only model, g processes only hypothesis embeddings: z=g(h\in H)=\text{ELECTRA}(h_{1},h_{2},\ldots,h_{n}) (1) Feeding z into a softmax layer produces the estimated probability distribution over output labels: \hat{p}=\mathrm{softmax}(U_{n}z+c) (2) where \mathrm{softmax}(r)_{j}=\frac{\exp(r_{j})}{\sum_{i}\exp(r_{i})}, U_{n}\in\mathbb{R}^{3\times d} is the weight matrix for the three NLI labels, and c is the bias vector. To train the hypothesis-only model, we minimize the cross-entropy loss function, which for an individual sample with target label y is given by: Loss(y)=-\sum_{j}y_{j}\log\hat{p}_{j} (3) This baseline highlights how models can achieve non-trivial accuracy without ever accessing premise information. Before we describe our debiased extension using Product-of-Experts, we take a quick detour to discuss why standard training on artifact-rich datasets is problematic. Connections to other debiasing frameworks are discussed further in Section 5. 3.2 The Problem with Standard Training Consider an example with premise “A person is sleeping” and hypothesis “Nobody is sleeping.” An artifact-based model could be deceived by the word “nobody” that returns a contradiction prediction without considering the premise. In contrast, debiased training objectives rely on downweighting such artifact-heavy examples during training, sacrificing some training efficiency in the process. This complexity is matched by important gains in model robustness. Standard training on biased datasets allows models to exploit spurious correlations that achieve high validation accuracy. As demonstrated by [ 18 ] , hypothesis-only models can reach 57.7\% accuracy on SNLI without accessing premise information. Our hypothesis-only baseline learns to map specific lexical patterns to labels: negation words (“not”, “never”) strongly correlate with contradiction, generic language correlates with neutral, and specific affirmative language with entailment. While standard models can achieve high benchmark accuracy by learning these shortcuts [ 5 , 4 ] , they suffer from critical weaknesses. The artifacts learned during training do not generalize to distribution shifts where these patterns differ. Models also fail on adversarial examples that maintain semantic meaning while violating learned surface patterns. Finally, standard training requires