In-Context Semi-Supervised Learning

Jiashuo Fan ∗, Paul Rosu ∗, Aaron T. Wang, Michael Li, Lawrence Carin, Xiang Cheng · 2025

Read In-Context Semi-Supervised Learning by Jiashuo Fan ∗, Paul Rosu ∗, Aaron T. Wang, Michael Li, Lawrence Carin, Xiang Cheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-15934.txt.

Introduction

Jiashuo Fan and co authors introduce in context semi supervised learning in a 2025 paper. Transformers learn in context from examples in their prompt, but most theory assumes every example is labeled. In practice labels are sparse, yet models still perform well, hinting that unlabeled context carries usable structure.

The authors formalize a setting where a few labeled examples come with many unlabeled points in the same context. They show transformers can use the unlabeled points to build a robust, context dependent representation. That representation enables accurate predictions and markedly better performance when labels are scarce. The work offers foundational insight into how transformers exploit unlabeled context.

Catalog record

CREATORJiashuo Fan ∗, Paul Rosu ∗, Aaron T. Wang, Michael Li, Lawrence Carin, Xiang Cheng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE100 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-15934

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

In-Context Semi-Supervised Learning Jiashuo Fan ∗, Paul Rosu ∗, Aaron T. Wang, Michael Li, Lawrence Carin, Xiang Cheng Abstract There has been significant recent interest on understanding the capacity of Transformers for in-context learning (ICL), yet most theory focuses on supervised settings with explicitly labeled pairs. In practice, Transformers often perform well even when labels are sparse or absent, suggesting crucial structure within unlabeled contextual demonstrations. We introduce and study in-context semi-supervised learning (IC-SSL) , where a small set of labeled examples is accompanied by many unlabeled points, and show that Transformers can leverage the unlabeled context to learn a robust, context-dependent representation. This representation enables accurate predictions and markedly improves performance in low-label regimes, offering foundational insights into how Transformers exploit unlabeled context for representation learning within the ICL framework. * * footnotetext: Equal contribution. 1 Introduction The Transformer ( 40 ) has revolutionized natural language processing, achieving remarkable success in tasks like language generation 40 ; 30 ; 16 ; 6 ; 37 ; 15 ; it has also been applied in many other areas, including vision 17 , network analysis 12 ; 23 ; 31 and biology 21 . Motivated by the demonstrated effectiveness of Transformers for few-shot (or in-context) learning 6 , a promising line of recent research has focused on functional data 41 ; 2 ; 43 ; 34 . In this setting one is given a set of contextual data pairs, where each pair is characterized by the observed input and output of a latent function. It has been demonstrated that a properly designed Transformer can in its forward pass perform functional gradient descent (GD) – or close variations thereof – for the function, and it can use that inference to predict the expected functional outcome for a specified query input to the function. The Transformer can perform this inference in-context, with no parameter fine-tuning. Early work on Transformer-based in-context learning (ICL) focused on linear functions 41 ; 2 ; 43 ; 34 , and that has been extended recently to nonlinear functions that reside in a reproducing kernel Hilbert space (RKHS) 11 . These works focus on learning real-valued outcomes, and similar analysis was recently developed for categorical outcomes 42 . Categorical observations are of interest when studying ICL for classification problems. Prior work on Transformer-based ICL largely treats the supervised case, using contextual labeled pairs. When unlabeled points appear, their labels are inferred independently, ignoring structure in the full unlabeled set. Yet in practice we have many unlabeled samples and few labels, and so it is of interest for ICL to leverage the context provided by all of this data. Figure 1 : Setup for semi-supervised in-context learning with a Transformer. Input Data consists of labeled pairs \{(x^{(1)},y^{(1)}),\dots,(x^{(m)},y^{(m)})\} , where y^{(i)}\in\{{{\color[rgb]{0,0,1}0}},{{\color[rgb]{1,0,0}1}}\} , and unlabeled data \{x^{(m+1)},\dots,x^{(n)}\} . In stage (A) , a Transformer module takes as input \{x^{(1)},\dots,x^{(n)}\} and outputs \phi(x^{(i)}) , where \phi(\cdot) is a context-dependent feature representation . In stage (B) , we augment \{\phi(x^{(1)}),\dots,\phi(x^{(n)})\} with the m observed labels . In stage (C) , a second Transformer module takes this augmented set as input. In stage (D) , the second Transformer module outputs prediction probabilities of y^{(m+1)},\dots,y^{(n)} for each unlabeled x^{(i)} in standard ICL fashion. The two Transformer modules combine into a single Transformer, trained end-to-end. As detailed in Section 3 , the first module is motivated by learning an eigenmap of the Laplacian, and the second module performs ICL with categorical observations by implementing gradient descent at inference time. 1.1 Outline of Contributions In-context semi-supervised learning setup. We extend Transformer-based in-context learning to a semi-supervised setting, broadening the concept of “context” to leverage both labeled and unlabeled data within a unified end-to-end Transformer architecture (Figure 1 ). We demonstrate how Transformers can effectively perform semi-supervised ICL, using scarce labeled data alongside abundant unlabeled examples, without the need for separate preprocessing of inputs. A two-stage Transformer construction for representation learning + ICL. We construct a two-stage Transformer (Section 3 ). The first stage performs representation learning from unlabeled data by forming a discrete Laplacian from local Euclidean distances, then computing an Eigenmap; this yields context-dependent manifold structure in the forward pass (Section 3.1 ). The second stage (Section 3.2 ) is a construction for a Transformer - based ICL module that can be shown to perform in - context learning for categorical observations by implementing functional gradient descent in its forward pass. Mechanistically , each stage is capable of implementing an explicit algorithm (1. Laplacian construction + power iteration; 2. kernelized GD for in-context inference), providing a first step toward a transparent account of how attention and MLPs realize semi-supervised ICL computations. In practice, we instantiate both stages inside a single Transformer and train the model end-to-end with the IC-SSL objective in Eq. ( 4 ); the modular construction serves as a strong mechanistic inductive bias/initialization rather than a hard constraint, and joint optimization can deviate from and improve upon the strict two-stage algorithm when beneficial. In addition, the construction serves as a prototype for analyzing full end-to-end Transformers: it gives a modular reference representation against which we can compare baseline Transformers trained directly on data (e.g., via separation and alignment analyses in Section F.6 ). Mechanistic understanding of IC-SSL. Section F.6 examines our method across manifolds of increasing complexity — low - dimensional manifolds in \mathbb{R}^{3} (Figure 2 ), higher - dimensional product manifolds in \mathbb{R}^{15} , and a high - dimensional image manifold from Stable Diffusion v1.5 (Figure 5 ) — where our construction consistently outperforms strong baselines. We then use the construction as a reference model to interpret how an unconstrained, end - to - end Transformer learns to perform in-context semi-supervised learning on real data (ImageNet100). Our model serves as a reference point for what representations are learned by the standard Transformer (Table 1 ), as well as for understanding the phase-transitions in the standard Transformer’s learning curves (Figure 3 ). Inductive bias and learning. Our two-stage design imposes a concrete inductive bias—local, sparse attention that can effectively implement spectral feature learning (first stage) and gradient-based learning (second stage) This makes IC - SSL markedly more sample - efficient in low - data regimes. As shown in Section F.6 , across elementary manifolds in \mathbb{R}^{3} (Figure 2 ), higher - dimensional product manifolds in \mathbb{R}^{15} (Figure 4(c) ), and high - dimensional image manifolds (Figure 5 ), this bias translates into consistently higher accuracy than strong baselines, including variants that rely on offline Laplacian Eigenmaps, even with few labeled examples. The same structural bias also supports robust generalization: the model retains strong in - distribution performance and transfers to previously unseen geometries (Figure 4(b) ) and modalities (Figure 5 ), indicating that it learns geometry - aware computations rather than overfitting to a specific dataset or coordinate system. Figure 2 : Synthetic manifolds we use. Left to right: Sphere (\mathbb{S}^{2}) , Right Circular Cylinder (\text{C}) , Right Circular Cone (\text{Cone}_{\alpha}) , Archimedean Spiral ( Swiss-Roll, SR ), Flat Torus (\mathbb{T}^{2}) . The colors reflect the binary labels. 1.2 Related Work Semi-supervised learning (SSL) has been a problem of long-standing interest to the machine learning community 39 ; 27 ; 8 . It is particularly valuable

Read the complete text