From Cross-Task Examples to In-Task Prompts: A Graph-Based Pseudo-Labeling Framework for In-context Learning
Zihan Chen, Song Wang, Xingbo Fu, Chengshuai Shi, Zhenyu Lei, Cong Shen, Jundong Li · 2025
Read From Cross-Task Examples to In-Task Prompts: A Graph-Based Pseudo-Labeling Framework for In-context Learning by Zihan Chen, Song Wang, Xingbo Fu, Chengshuai Shi, Zhenyu Lei, Cong Shen, Jundong Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24528.txt.
Introduction
Zihan Chen and six co-authors present a graph based pseudo-labeling framework that makes in context learning cheaper for new tasks. Language models can learn a task from a few examples in the prompt, but collecting quality examples for hard or novel tasks is costly. Their two stage pipeline reduces reliance on the model for labeling.
First, readily available examples from other tasks prompt the model to pseudo-label a small set of target instances. Then a graph based label propagation spreads those labels to the remaining target examples with no further model queries. The fully pseudo-labeled set becomes in task prompts, cutting labeling cost while supporting in context learning.
Catalog record
| CREATOR | Zihan Chen, Song Wang, Xingbo Fu, Chengshuai Shi, Zhenyu Lei, Cong Shen, Jundong Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-24528
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Cross-Task Examples to In-Task Prompts: A Graph-Based Pseudo-Labeling Framework for In-context Learning Zihan Chen, Song Wang, Xingbo Fu, Chengshuai Shi, Zhenyu Lei, Cong Shen, Jundong Li Abstract The capability of in-context learning (ICL) enables large language models (LLMs) to perform novel tasks without parameter updates by conditioning on a few input-output examples. However, collecting high-quality examples for new or challenging tasks can be costly and labor-intensive. In this work, we propose a cost-efficient two-stage pipeline that reduces reliance on LLMs for data labeling. Our approach first leverages readily available cross-task examples to prompt an LLM and pseudo-label a small set of target task instances. We then introduce a graph-based label propagation method that spreads label information to the remaining target examples without additional LLM queries. The resulting fully pseudo-labeled dataset is used to construct in-task demonstrations for ICL. This pipeline combines the flexibility of cross-task supervision with the scalability of LLM-free propagation. Experiments across five tasks demonstrate that our method achieves strong performance while lowering labeling costs. Our code is available at https://github.com/Chen-1031/Cross-Task-ICL . 1 Introduction Large language models (LLMs) have demonstrated impressive capabilities across a wide range of natural language processing tasks Zhao et al. 2023 ; Chang et al. 2024 , including semantic parsing Li et al. 2021 ; Wolfson et al. 2020 and commonsense reasoning Talmor et al. 2019 ; Zellers et al. 2019 ; Lei et al. 2025b ; Lei et al. 2025a . However, the substantial computational cost of retraining or fine-tuning these models limits their practicality for novel tasks Hu et al. 2021 ; Liu et al. 2022 ; Zaken et al. 2022 . Fortunately, LLMs possess an emergent ability known as In-Context Learning (ICL) Wang et al. 2024c ; Wang et al. 2024a ; Chen et al. , wherein the model can perform new tasks by conditioning on a few input-output pairs (i.e., demonstrations) during inference, without updating model parameters Brown et al. 2020 . Task Label Space QQP {duplicate, not duplicate} Financial-Phrasebank {positive, neutral, negative} Figure 1: (Top) Wasserstein distance between source (column) and target (row) task example embeddings. (Bottom) Examples of task label spaces. Despite its promise, the effectiveness of ICL heavily relies on high-quality labeled examples for the target task. For novel or data-scarce tasks, recent work has explored using LLMs with zero-shot prompts Zhang et al. 2025 ; Wan et al. 2024 ; Shi et al. 2024 or relying on human annotators Mikulová et al. 2023 ; Wang et al. 2024b to obtain pseudo-labeled examples, which are then used as demonstrations for ICL. Yet both approaches have drawbacks: LLMs can be unreliable on unfamiliar tasks, while human annotation introduces additional time and labor costs Su et al. 2022a . To overcome this, recent efforts have turned to leveraging well-established, high-resource source tasks to construct demonstrations Tanwar et al. 2023 ; Raffel et al. 2020 . When the examples used for ICL are drawn from a different task than the target, the setting is referred to as cross-task ICL. For example, Chatterjee et al. 2024 select examples from source tasks based on embedding similarity and demonstrate that such cross-task examples can significantly improve ICL performance, highlighting cross-task ICL as a promising approach for pseudo-labeling. However, as illustrated in Figure 1 , the data distributions of source and target tasks often differ significantly, and the label spaces can be misaligned, even between tasks with similar distributions. This raises a key limitation: selecting cross-task examples based solely on embedding similarity is insufficient for reliable pseudo-labeling of target samples. To address this, we draw inspiration from the graph mining literature, which shows that structural properties of graphs can generalize across domains even when feature spaces are heterogeneous Qiu et al. 2020 ; Leskovec et al. 2005 ; Hamilton et al. 2017 . Based on this, we propose GraphSim, a graph-based example selection method that augments text embeddings with structural information captured through graph aggregation—a process where each node’s representation is updated by aggregating information from its neighbors. These structure-aware embeddings yield more robust similarity metrics across tasks, allowing for better example selection. To further address the label space mismatch across tasks, we propose GLIP (Graph-based Label Information Propagation), a label information propagation framework that uses a small set of pseudo-labeled target examples (obtained via GraphSim and an LLM) to infer labels for the remaining unlabeled target examples. The resulting pseudo-labeled target set can then be used as high-quality demonstrations for in-context learning. Notably, our pipeline is cost-efficient and adaptable: it only requires a small number of LLM calls for pseudo-labeling and leverages lightweight graph-based propagation for the rest, making it practical for real-world deployment on novel tasks. Our contributions can be summarized as follows: • Problem Formulation : We propose a novel problem formulation that utilizes examples from high-resource source tasks to pseudo-label examples from a novel target task. This enables in-context learning without requiring extensive manual annotation or large-scale LLM inference on the target task. • Methodological Innovation : We introduce a two-stage graph-based pipeline that addresses both cross-task example selection and label propagation. First, we propose GraphSim, a structure-aware similarity metric for selecting source examples that are more transferable across tasks. Second, we design GLIP, which efficiently propagates labels from a few LLM-labeled target examples to the rest of the target dataset, mitigating label space misalignment. • Empirical Validation : Comprehensive experiments on five target tasks with five different-sized LLMs demonstrate that our method outperforms existing cross-task baselines and approaches the performance of in-task upper bounds with light reliance on LLMs, highlighting both its effectiveness and efficiency. 2 Related Works In-Context Learning. In-context learning (ICL) Brown et al. 2020 equips large language models (LLMs) with the ability to leverage a handful of input-output demonstrations for reasoning. ICL has proven remarkably successful in handling complex tasks, including summarization Jain et al. 2023 ; Baek et al. 2024 and question answering Lee et al. 2024 . To improve ICL performance, many works have explored adaptive strategies for selecting effective demonstrations Lu et al. 2021 ; Zhao et al. 2021 ; Ye et al. 2023 ; Chen et al. 2024 , such as retrieving semantically similar examples Liu et al. 2021 . However, these methods typically assume that the demonstrations and the input come from the same task (i.e., in-task ICL) and rely on access to a large pool of labeled examples. This assumption limits their applicability in low-resource or novel task settings, where collecting high-quality annotations is costly and impractical. Cross-task In-Context Learning. Due to the large disparity in annotation availability between novel tasks and well-established ones, leveraging high-resource source tasks to improve performance on low-resource target tasks has become an appealing direction. Prior work has explored cross-task ICL in various settings, such as cross-lingual tasks Tanwar et al. 2023 , multi-task learning Zhang et al. 2022 ; Raffel et al. 2020 , and prompt generation for downstream tasks Zou et al. 2023 . More recently, Chatterjee et al. 2024 demonstrated that LLMs can benefit significantly from cross-task ICL prompts and showed the potential of generating pseudo-labels for in-task examples. However, their method relies on semantic similarity for example selection and labels only a small subset of target examples, leaving most unlabeled