Hidden Biases in Conditioning Autoregressive Models

François Pachet, Pierre Roy · 2026

Read Hidden Biases in Conditioning Autoregressive Models by François Pachet, Pierre Roy (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-07855.txt.

Description

Pachet and Roy expose a hidden bias in constrained generation from autoregressive models, used for rhyming lines, fixed meter, infilling, and positional endings. These systems often look strong, yet their sampling procedures are not exact conditioning of the underlying model. Samples therefore skew away from the true constrained distribution, with no guarantee of covering all valid completions or weighting them correctly. The paper formalizes several exact inference tasks for such models and proves matching hardness results, separating this inferential bias from bias inherited from training data.

Catalog record

CREATORFrançois Pachet, Pierre Roy
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25432
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.07855

Browse the full library · About the Tower of Babel