A Reproducibility Analysis of PO4ISR: Diagnosing and Mitigating Semantic Drift in LLM-Based Session Recommendation
Aditya Tiwari, Konduri Naga Lakshmi Rekha*, Rajesh Kumar Mundotiya · 2026
Read A Reproducibility Analysis of PO4ISR: Diagnosing and Mitigating Semantic Drift in LLM-Based Session Recommendation by Aditya Tiwari, Konduri Naga Lakshmi Rekha*, Rajesh Kumar Mundotiya (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18780.txt.
Introduction
Aditya Tiwari, Konduri Naga Lakshmi Rekha, and Rajesh Kumar Mundotiya conduct, in a 2026 analysis, a reproducibility study of PO4ISR. Standard reasoning prompts suffer severe contextual drift in long sessions, hurting performance on Games and Bundle. Their PO4ISR-Plus variant adds reflexive prompting and consistent rank detection for cross-domain cues. Benchmarked on ML-1M, Games, and Bundle, it restores performance, with gains up to fifty-four percent on Games and ninety-six on Bundle. Open-source artifacts include the reproduced baseline and the enhanced framework.
Catalog record
| CREATOR | Aditya Tiwari, Konduri Naga Lakshmi Rekha*, Rajesh Kumar Mundotiya |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35194 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.18780
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Reproducibility Analysis of PO4ISR: Diagnosing and Mitigating Semantic Drift in LLM-Based Session Recommendation Aditya Tiwari, Konduri Naga Lakshmi Rekha*, Rajesh Kumar Mundotiya Abstract Reasoning-based Large Language Models (LLMs) like PO4ISR have set new benchmarks in session-based recommendation. However, the reproducibility of their reasoning capabilities across diverse semantic domains remains unexplored. In this work, we conduct a rigorous reproducibility study of PO4ISR to assess its generalization limits. Our analysis reveals a critical failure mode: standard reasoning prompts suffer from severe contextual drift in long sessions, leading to performance degradation on semantically complex datasets like Games and Bundle. To quantify and resolve this stability gap, we introduce PO4ISR++, a robustness-enhanced implementation that integrates reflexive prompting and consistent rank detection. Unlike the original static prompting strategy, our approach dynamically adapts to cross-domain cues. We benchmark both the original implementation and our robust variant on ML-1M, Games, and Bundle. Our results confirm that while the original model struggles in new domains, our reproducible extension restores performance, yielding a stabilized gain of up to 54% on Games and 96% on Bundle. We release open-source artifacts, including the reproduced baseline and our enhanced framework, to facilitate reliable future research in LLM-based recommendation. Keywords Session-based Recommendation, Large Language Models (LLMs), Recommender Systems, Cross-Domain Generalization, Reproducibility, 1 Introduction Session-based recommendation (SR) has established itself as a cornerstone of modern information retrieval, adhering to the principle of privacy by predicting user preferences based solely on short-term interaction sequences 17 ; 11 ; 31 . Unlike conventional collaborative filtering, which relies on persistent user profiles, SR systems must infer evolving user intent from sparse, transient behavioral cues-a challenge critical for anonymous environments in e-commerce and media streaming. While early approaches leveraged Recurrent Neural Networks (RNNs) 5 ; 25 and Graph Neural Networks (GNNs) 20 ; 29 to model sequential dependencies, recent advancements have shifted toward the reasoning capabilities of Large Language Models (LLMs) 6 ; 32 . Notable frameworks such as PO4ISR 23 demonstrate that LLMs can extract deep semantic insights via prompt engineering, offering a promising avenue for interpretable recommendation. However, the transition from latent vectors to prompt-based reasoning introduces new barriers to system reliability and reproducibility. In reproducing state-of-the-art LLM-based approaches, we identify two critical failure modes that hinder practical deployment: 1. Stochastic Output Instability: Current methods often rely on text-generation to predict item indices. We observe that this approach degrades significantly when item metadata contains numerical ambiguity (e.g., "Goya Foods Mango Nectar, 5-Ounce [Pack of 48 ]" or "Xbox 360 " ). The LLM frequently misinterprets these quantities as ranking indices, leading to hallucinated outputs and parsing failures that sever the link between reasoning and recommendation. 2. Domain Overfitting via Static Prompting: Existing strategies typically employ a “Top-1” prompt optimization approach 23 , where a prompt tuned for a specific dataset is applied universally. Our analysis reveals that this results in domain-specific overfitting, failing to generalize to datasets with distinct semantic structures (e.g., moving from Movies to Bundles). These limitations highlight a gap between the theoretical potential of reasoning-based SR and its reproducible robustness across diverse real-world domains. 1.1 Contributions To bridge this gap, we present a reproducibility study and robustness enhancement of the reasoning-based SR paradigm. We introduce PO4ISR ++ , a reflexive deterministic framework designed to ensure consistent performance across domains. Our specific contributions are as follows: • Diagnostic Reproducibility Analysis: We quantify the stability gap in existing LLM-SR models, systematically categorizing failure cases caused by numerical ambiguity and prompt stochasticity. • Deterministic Rank-Mapping: We propose an index-specific output formatting strategy that decouples semantic reasoning from syntax generation. This eliminates parsing ambiguity, ensuring valid, reproducible rankings even for items with complex numerical nomenclature. • Reflexive Cross-Domain Fusion: Leveraging the meta-reasoning capabilities of Gemini-2.0, we introduce a reflexive prompting strategy. Unlike static methods, this approach fuses insights from multi-domain training (ML-1M, Games, Bundle) to discover generalizable prompt patterns that transfer effectively across disparate semantic contexts. • Robustness Benchmarking: Extensive experiments demonstrate that PO4ISR ++ not only resolves the parsing failures of its predecessor but also achieves statistically significant performance gains, establishing a new, robust baseline for the reproducible evaluation of LLM-based recommenders. 2 Related Work 2.1 Evolution of Session-Based Recommendation Session-Based Recommendation (SR) has progressed through three distinct modeling paradigms: sequential, graph-based, and attention-driven architectures. Early sequential approaches pioneered by Hidasi et al. 5 utilized Gated Recurrent Units (GRUs) to model interaction sequences. Subsequent variants integrated attention mechanisms (NARM 11 , STAMP 17 ) to capture transient user interests. However, these recurrent architectures often struggled with long-range dependencies and the rigid ordering of interactions. To address these structural limitations, Graph Neural Networks (GNNs) emerged as a powerful alternative. Wu et al. 29 introduced SR-GNN to model complex, non-consecutive item transitions, inspiring extensions that incorporated target-aware propagation 30 and global context 28 . Concurrently, the success of Transformers in NLP led to the adoption of self-attention mechanisms in recommendation (SASRec 8 , BERT4Rec 22 ), which enabled parallelized processing and superior modeling of global dependencies. Despite their predictive success, these latent-based models face a common limitation: they operate as black boxes, mapping behaviors to opaque vector spaces without explicit reasoning, often failing to capture the multiplicity of conflicting user intents within a single session 10 ; 26 . 2.2 Intent-Aware & Multi-Interest Modeling To enhance interpretability and granularity, Intent-Aware Session Recommendation (ISR) focuses on explicitly modeling the latent goals driving user behavior. Early ISR methods assumed a monolithic intent per session 2 ; 19 , which proved insufficient for dynamic browsing behaviors. More recent approaches, such as MCPRN 15 and HIDE 14 , employ multi-channel networks to disentangle concurrent user goals. Contrastive learning frameworks 3 ; 13 further refine this by maximizing the distinctiveness between different intent representations. While these methods improve the granularity of representations, they remain bound to latent embeddings, lacking the semantic transparency required for explainable recommendation. 2.3 LLMs and Reproducibility Challenges in SR The integration of Large Language Models (LLMs) marks a paradigm shift from “matching” to “reasoning.” Unlike collaborative filtering, LLM-based approaches leverage vast pre-trained knowledge to perform zero-shot or few-shot recommendation (TALLRec 1 , GPT4Rec 12 ). However, this shift introduces significant reproducibility and reliability challenges. Empirical studies 9 ; 16 indicate that while LLMs excel at general reasoning, they suffer from stochastic instability and ranking inconsistency when applied to specific recommendation tasks. Geng et al. 4 attempted to mitigate this via unified pretrain-prompt-predict frameworks, yet domain adaptation remains a bottleneck. Most relevant to our work is PO4ISR 23 , which established a state-of-the-art baseline by applying prompt optimization to ISR tasks. While PO4ISR demonstrated the efficacy of reasoning-based recommendations, its original evaluation relied on static, domain-specific prompts and unstructured generation. As we demonstrate in this study, these design choices lead to fragile generalization–specifically, vulnerability to numerical ambiguity and domain drift. Our work directly addresses these gaps, moving beyond simple application to rigorously benchmark and enhance the reproducibility of reasoning-based ISR. 3 PO4ISR ++ : Enhancing Reproducibility via Determinism and Fusion 3.1 Problem Formulation We frame SR as a prompt optimization task with a ranking objective. Let \mathcal{S} denote a set of user sessions, where each s\in\mathcal{S} is a sequence of interactions. Given a candidate pool \mathcal{C} and a ground-truth