When and How Unlabeled Data Provably Improve In-Context Learning

Yingcong Li, Xiangyu Chang, Muti Kara, Xiaofeng Liu, Amit Roy-Chowdhury, Samet Oymak · 2025

Read When and How Unlabeled Data Provably Improve In-Context Learning by Yingcong Li, Xiangyu Chang, Muti Kara, Xiaofeng Liu, Amit Roy-Chowdhury, Samet Oymak (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15329.txt.

Introduction

Yingcong Li and co-authors prove when unlabeled examples help a model learn from context, in a 2025 paper. In-context learning lets a model pick up a task from examples in its prompt. Oddly, it can work even when some example labels are missing or wrong. The team studied a clean lab case: prompt examples drawn from two blended groups, with a share of labels blank.

Simple one-layer attention models, they show, find the best fully labeled answer but gain nothing from the blank cases. Deeper or looped models can use them. Such models build, step by step, a formula that blends the data with the known labels, much like a classic fix-up method for missing labels. A bit of depth buys a lot of power. Looping a stock table model the same way lifted its scores on real data.

Catalog record

CREATORYingcong Li, Xiangyu Chang, Muti Kara, Xiaofeng Liu, Amit Roy-Chowdhury, Samet Oymak
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE100 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-15329

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When and How Unlabeled Data Provably Improve In-Context Learning Yingcong Li, Xiangyu Chang, Muti Kara, Xiaofeng Liu, Amit Roy-Chowdhury, Samet Oymak Abstract Recent research shows that in-context learning (ICL) can be effective even when demonstrations have missing or incorrect labels. To shed light on this capability, we examine a canonical setting where the demonstrations are drawn according to a binary Gaussian mixture model (GMM) and a certain fraction of the demonstrations have missing labels. We provide a comprehensive theoretical study to show that: (1) The loss landscape of one-layer linear attention models recover the optimal fully-supervised estimator but completely fail to exploit unlabeled data; (2) In contrast, multilayer or looped transformers can effectively leverage unlabeled data by implicitly constructing estimators of the form ∑ i ≥ 0 a i ​ ( 𝑿 ⊤ ​ 𝑿 ) i ​ 𝑿 ⊤ ​ 𝒚 \sum_{i\geq 0}a_{i}({\bm{X}}^{\top}{\bm{X}})^{i}{\bm{X}}^{\top}\bm{y} with 𝑿 {\bm{X}} and 𝒚 \bm{y} denoting features and partially-observed labels (with missing entries set to zero). We characterize the class of polynomials that can be expressed as a function of depth and draw connections to Expectation Maximization, an iterative pseudo-labeling algorithm commonly used in semi-supervised learning. Importantly, the leading polynomial power is exponential in depth, so mild amount of depth/looping suffices. As an application of theory, we propose looping off-the-shelf tabular foundation models to enhance their semi-supervision capabilities. Extensive evaluations on real-world datasets show that our method significantly improves the semisupervised tabular learning performance over the standard single pass inference. 1 Introduction In-context learning (ICL) is an intriguing capability of modern language models and has enjoyed remarkable empirical success ( Brown et al. 2020 ; Min et al. 2022 ) . This success is also being extended to multimodal scenarios ( Zhou et al. 2024 ) as well as other modalities such as tabular data ( Hollmann et al. 2022 ) . The push toward test-time scaling and long-context models ( Snell et al. 2024 ; Guo et al. 2025 ) has further boosted the benefits of ICL by allowing the model to ingest a large number of demonstrations. For instance, in “Many-shot in-context learning” paper, Agarwal et al. 2024 demonstrate that pushing more examples into context window can substantially boost the accuracy. MAPLE ( Chen et al. 2025 ) improves many-shot ICL by pseudo-labeling high-impact unlabeled examples and incorporating them into the prompt. The many-shot ICL setting naturally raises the question of when and how ICL can succeed with weaker supervision. As we can harness longer context models to boost predictive accuracy, we may indeed run out of high-quality demonstrations with verified answers/chain-of-thoughts and may want to utilize weaker data sources. This motivates our central question: \emph{\textbf{Q:}~Whenandhowcantransformerslearnincontextfromunlabeleddata?} We primarily investigate this question under a semisupervised ICL (SS-ICL) setting with Gaussian mixture models (GMMs). Formally, given a prompt containing a dataset of feature-label pairs (\bm{x}_{i},y_{i})_{i=1}^{n}\in\mathbb{R}^{d}\times\mathbb{R} as demonstrations and a query feature \bm{x} (see Eq. ( 3 )), a model trained for ICL learns to predict the corresponding output y given prompt. For ICL with a supervised binary GMM model, we have \bm{x}_{i}\sim{\cal{N}}({\bm{\mu}}_{y_{i}},\sigma^{2}{\bm{I}}) and y_{i}\in\{-1,1\} , i\in[n] , and the component means {\bm{\mu}}_{\pm 1} that parameterize the classification task are sampled from a prior task distribution. This prompt model is well studied under various fully-supervised settings ( Garg et al. 2022 ; Von Oswald et al. 2023 ; Ahn et al. 2023 ; Akyürek et al. 2023 ; Mahankali et al. 2024 ; Collins et al. 2024 ; Shen et al. 2024 ) where each demonstration includes a clearly labeled output. In our SS-ICL setting, only m out of n total samples have correct labels ( m\leq n ) either -1 or 1 , and remaining labels are unknown and fed to the model as y_{i}=0 . In this work, we provide a comprehensive theoretical and empirical study of attention models with varying depths when trained with SS-ICL. Our analysis reveals the importance of depth : Despite being able to implement the optimal fully-supervised estimator, single-layer linear attention completely fails to leverage unlabeled examples. In contrast, deeper or looped transformer architectures can emulate strong semi-supervision algorithms, approaching the performance of the Bayes-optimal classifier as depth increases. Informed by the importance of depth/looping, we also devise semisupervision strategies for tabular foundation models. Our specific contributions are: \diamond Landscape of one-layer linear attention ( \lx@sectionsign 3 ): We study the optimization landscape of single-layer linear attention for the SS-ICL problem under an isotropic task prior. We prove that the global minimum of the loss function returns the plug-in estimator (see Eq. ( SPI )), i.e., \hat{y}=\textrm{sgn}(\bm{x}^{\top}\hat{{\bm{\mu}}}) with \hat{{\bm{\mu}}}={\bm{X}}^{\top}\bm{y} , where {\bm{X}}\in\mathbb{R}^{n\times d} represents features and \bm{y}\in\mathbb{R}^{n} denotes partially-observed labels (with missing entries set to zero) of the ICL demonstrations. This implies that 1-layer model learns Bayes-optimal classifier in the fully-supervised setting, but completely fails to make use of unlabeled data. \diamond Depth is crucial but shallow can suffice (§ 4 ): We show that multilayer linear attention can emulate semisupervised learners by implementing polynomial estimators of the form \displaystyle\hat{{\bm{\mu}}}=\sum_{i=0}^{K}a_{i}({\bm{X}}^{\top}{\bm{X}})^{i}{\bm{X}}^{\top}\bm{y}. (1) Crucially, an L -layer (or looped) attention can express up to K=O(3^{L}) powers, highlighting that logarithmic depth suffices to represent high-degree monomials. We provide characterizations of the set of expressible polynomials through different constructions (where each layer gets to update the features or labels of the previous layer). Corroborating these, experiments reveal that shallow models with L\geq 2 already achieve strong results and their performance can be approximately predicted through an eigen-estimator combining i=0 and \infty (see ( SSPI- k )). \diamond What learner attention emulates? In Section 4.3 , we describe how each attention block can update the label estimates by emulating expectation-maximization (for linear attention) or belief propagation (for softmax attention). For instance ( 1 ) can be interpreted as the model implicitly conducting an Expectation-Maximization algorithm: Starting with the supervised estimator \bm{\hat{\mu}}_{0}={\bm{X}}^{\top}\bm{y} , each term ({\bm{X}}^{\top}{\bm{X}})^{i}{\bm{X}}^{\top}\bm{y} can be viewed as a sequence of pseudo-labeling (expectation) \hat{\bm{y}}_{i}={\bm{X}}\bm{\hat{\mu}}_{i-1} and training (maximization) \bm{\hat{\mu}}_{i}={\bm{X}}^{\top}\hat{\bm{y}}_{i} steps. Corroborating this, we show that softmax-attention and softmax-transformer models similarly benefit from increasing depth and can emulate semisupervised learners competitive with Bayes limit (see Fig. 2(c) ). \diamond Applications to Tabular FMs (§ 5 ): Tabular foundation models such as TabPFN ( Hollmann et al. 2022 ; Hollmann et al. 2025 ) and TabICL ( Qu et al. 2025 ) represent a suitable application of theory as they also model the ICL examples with a single token. To harness unlabeled examples, we propose a novel strategy that iteratively creates soft pseudo-labels by explicitly looping the tabular FM while controlling validation risk. Focusing on the few-shot learning setting where TabPFN-v2 ( Hollmann et al. 2025 ) excels, we demonstrate that our approach can significantly improve predictive performance on various real-world datasets. 1.1 Related Work Theoretical Analysis of In-Context Learning Recent work has developed theoretical frameworks for understanding in-context learning in transformers. Akyürek et al. 2023 , Von Oswald et al. 2023 and Dai et al. 2023 demonstrated that transformers emulate gradient descent during ICL. Xie et al. 2022 offered a Bayesian perspective, while Zhang et al. 2023 ; Zhang et al. 2024 showed transformers learn linear models in-context. Ahn et al. 2023 established they implement preconditioned gradient descent, and Mahankali et al. 2024 proved one-step

Read the complete text