Architectural and Inferential Inductive Biases For

2025

Read Architectural and Inferential Inductive Biases For (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01215.txt.

Introduction

Daksh Mittal and co-authors study how sequence models should handle uncertainty in repeated observations, in work from 2025. Exchangeable sequences arise wherever data points share a hidden common cause. Posterior reasoning over that cause drives tasks like active learning and bandits.

Single-step generation, the common habit, cannot separate two kinds of uncertainty, they show. Multi-step generation, long favored by Bayesian statisticians, fixes that flaw. Better uncertainty scores then improve downstream decisions in their tests. Architecture choice for multi-step inference leaves a subtle gap, though. Recent transformer rivals are compared on that point. The paper guides design of models that must reason, not just predict.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-01215

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Architectural and Inferential Inductive Biases For Abstract Autoregressive models have emerged as a powerful framework for modeling exchangeable sequences—i.i.d. observations when conditioned on some latent factor—enabling direct modeling of uncertainty from missing data (rather than a latent). Motivated by the critical role posterior inference plays as a subroutine in decision-making (e.g., active learning, bandits), we study the inferential and architectural inductive biases that are most effective for exchangeable sequence modeling. For the inference stage, we highlight a fundamental limitation of the prevalent single-step generation approach: inability to distinguish between epistemic and aleatoric uncertainty. Instead, a long line of works in Bayesian statistics advocates for multi-step autoregressive generation; we demonstrate this "correct approach" enables superior uncertainty quantification that translates into better performance on downstream decision-making tasks. This naturally leads to the next question: which architectures are best suited for multi-step inference? We identify a subtle yet important gap between recently proposed Transformer architectures for exchangeable sequences [ 22 , 23 , 30 ] , and prove that they in fact cannot guarantee exchangeability despite introducing significant computational overhead. We illustrate our findings using controlled synthetic settings, demonstrating how custom architectures can significantly underperform standard causal masks, underscoring the need for new architectural innovations. Exchangeable Sequence Modeling Daksh Mittal ∗ Ang Li ∗ Tzu-Ching Yen ∗ Daniel Guetta Hongseok Namkoong Decision, Risk, and Operations Division, Columbia Business School {dm3766, al4263, ty2531, crg2133, hongseok.namkoong}@columbia.edu * * footnotetext: Equal contribution 1 Introduction Intelligent agents must be able to articulate their own uncertainty about the underlying environment, and sharpen its beliefs as it gathers more information. We consider an sequence of observations Y_{1:\infty} gathered from an unseen environment \theta , e.g., noisy answers in a math quiz, generated by a student’s current proficiency level \theta . When marginalizing over the latent \theta , the joint distribution of the sequence Y_{1:\infty} is permutation invariant, and thus exchangeable sequences form a basic unit of study in uncertainty quantification of latents that govern data generation. Autoregressive sequence modeling has recently gained significant attention as a powerful approach for modeling exchangeable sequences Y_{1:\infty} [ 22 , 23 , 31 , 30 , 20 ] . Unlike conventional Bayesian modeling—which requires specifying a prior over an unobserved latent variable \theta and a likelihood for the observed data, often a challenging task—autoregressive sequence modeling builds on De Finneti’s predictive view of Bayesian inference [ 6 , 7 , 8 , 9 ] . This view directly models the observables Y_{1:\infty} : for an exchangeable sequence, epistemic uncertainty in the latent variable \theta stems from the unobserved future data [ 2 , 4 , 12 ] . Viewing future observations as the sole source of epistemic uncertainty in \theta for exchangeable sequences [ 2 , 4 , 12 ] , autoregressive sequence modeling enables direct prediction of the observables Y_{1:\infty} , offering a conceptually elegant and practical alternative to conventional Bayesian approaches. Transformers have emerged as the dominant architecture for autoregressive sequence modeling [ 20 , 22 , 17 , 18 , 15 , 29 ] , owing to their remarkable performance in natural language and vision applications [ 5 , 10 ] . As Transformers are increasingly employed to meta-learn probabilistic models for large-scale tabular datasets [ 32 , 20 ] , they offer a unique opportunity to move beyond traditional prediction tasks or merely replicating supervised algorithms—an area that has been the primary focus so far. Instead, following De Finetti’s perspective, when meta-trained on tabular datasets, these models can effectively quantify epistemic uncertainty, which powers decision-making and active exploration across diverse domains, including recommendation systems, adaptive experimentation, and active learning [ 31 ] . For instance, we can train sequence models on a collection of tables, each representing a different disease diagnosis setting—akin to applying meta-learning in the context of disease diagnosis. These models can then be leveraged to actively gather additional data in a previously unseen disease diagnosis setting to enhance model’s predictive performance (see Figure 2 for illustratation). However, using Transformers to model exchangeable sequences for decision-making presents its own challenges. Since the existing literature has primarily focused on traditional prediction tasks or the replication of supervised algorithms rather than decision-making, it has overlooked the perspective that epistemic (reducible) uncertainty stems from missing data. Accurately distinguishing between epistemic (reducible) and aleatoric (irreducible) uncertainty is crucial for decision-oriented applications. A significant limitation in the current literature is its predominant focus on one-step predictive uncertainty [ 22 , 23 ] or one-step predictions [ 18 ] , which fails to differentiate between epistemic and aleatoric uncertainty. In contrast, multi-step inference provides a more robust framework for differentiating between epistemic and aleatoric uncertainty [ 28 , 24 ] (see Figure 3 to come for an illustrative example). Through both empirical analysis and theoretical examination, we demonstrate impact of one-step and multi-step inference on uncertainty quantification (Theorem 2 , Figure 4 ) and downstream decision-making tasks, particularly in multi-armed bandit and active learning settings. Our findings show that multi-step inference consistently outperforms one-step inference in these tasks (see Section 3 ). These results align with those of Zhang et al. 2024 , who also proposed multi-step inference to enhance active exploration via Thompson sampling. Figure 1 : Equivalence between conventional Bayesian modeling and predictive view of Bayesian inference is established by De Finneti (valid only under infinite exchangeability of Y_{1:\infty} ). It establishes that epistemic uncertainty in latent parameter \theta\sim\mu(\cdot|y_{1:t}) is equivalent to predictive uncertainty of future observations Y_{t+1:\infty}\sim P(\cdot|y_{1:t}) . This brings us to the next key question: What kind of architectures should we use for modeling exchangeable sequences, particularly when performing multi-step inference? Architectural choice is a crucial source of inductive bias, and existing approaches have attempted to incorporate exchangeability as an inductive bias through specialized masking strategies [ 22 , 23 , 30 ] . However, a critical gap in the literature remains: these prior works conflate exchangeability with the invariance properties enforced by their masking schemes that don’t necessarily guarantee valid probabilistic inference. A key contribution of our work is in articulating this gap. We introduce conditional permutation invariance (Property 1 ) as a way to formalize the invariance enforced by these masking strategies. Correcting previous works [ 22 , 23 , 30 ] that implicitly assume their architectures achieve exchangeability, we show that enforcing conditional permutation invariance alone is insufficient to guarantee full exchangeability. Specifically, existing masking-based approaches do not ensure the conditionally identically distributed (c.i.d.) property (Property 2 ), which is essential for the validity of probabilistic inference in exchangeable sequence models. As a result, despite their intended design, such models may still violate exchangeability when used in practice (see Section 4 ). By clearly distinguishing between exchangeability and the weaker notion of conditional permutation invariance, our work not only clarifies existing misconceptions in the literature but also establishes a more rigorous foundation for developing exchangeable sequence models. Figure 2 : Meta-learned sequence models can be used for decision making Moreover, a significant drawback of this masking scheme, as discussed in Section 4 , is that it introduces computational overhead without yielding any tangible improvements in model performance. To evaluate its impact, we empirically assess the effectiveness of enforcing conditional-permutation invariance (Property 1 ) within the

Read the complete text