BERAG: Bayesian Ensemble Retrieval-Augmented Generation for Knowledge-based Visual Question Answering

Jinghong Chen, Jingbiao Mei, Guangyu Yang, Bill Byrne · 2026

Read BERAG: Bayesian Ensemble Retrieval-Augmented Generation for Knowledge-based Visual Question Answering by Jinghong Chen, Jingbiao Mei, Guangyu Yang, Bill Byrne (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22678.txt.

Introduction

Chen and three colleagues propose BERAG and BEFT, Bayesian ensemble retrieval-augmented generation and fine-tuning, for knowledge-based visual question answering. The usual retrieval-augmented generation joins documents into one context, hiding each document's role, hurting attribution, causing the lost-in-the-middle effect where key long-context facts are missed. Joining texts scales poorly, with square-law cost growth, worst for visual data. Capping context length can still curb performance by keeping models from using deeper retrieval recall. In BERAG, language models use single retrieved documents.

Catalog record

CREATORJinghong Chen, Jingbiao Mei, Guangyu Yang, Bill Byrne
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41411
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.22678

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

BERAG: Bayesian Ensemble Retrieval-Augmented Generation for Knowledge-based Visual Question Answering Jinghong Chen, Jingbiao Mei, Guangyu Yang, Bill Byrne Abstract A common approach to question answering with retrieval-augmented generation (RAG) is to concatenate relevant documents into a single, large context and pass it to a language model to generate an answer. While simple, this strategy can obscure the contribution of individual documents, making attribution difficult and contributing to the “lost-in-the-middle” effect, where relevant information in long contexts is overlooked. Concatenation also scales poorly: computational cost grows quadratically with context length, a problem that becomes especially severe when the context includes visual data, as in visual question answering. Attempts to mitigate these issues by limiting context length can further restrict performance by preventing models from benefiting from the improved recall offered by deeper retrieval. We propose Bayesian Ensemble Retrieval-Augmented Generation (BERAG), along with Bayesian Ensemble Fine-Tuning (BEFT), as a RAG framework in which language models are conditioned on individual retrieved documents rather than a single combined context. BERAG treats document posterior probabilities as ensemble weights and updates them token by token using Bayes’ rule during generation. This approach enables probabilistic re-ranking, parallel memory usage, and clear attribution of document contribution, making it well-suited for large document collections. We evaluate BERAG and BEFT primarily on knowledge-based visual question answering tasks, where models must reason over long, imperfect retrieval lists. The results show substantial improvements over standard RAG, including strong gains on Document Visual Question Answering and multimodal needle-in-a-haystack benchmarks. We also demonstrate that BERAG mitigates the “lost-in-the-middle” effect. The document posterior can be used to detect insufficient grounding and trigger deflection, while document pruning enables faster decoding than standard concatenative RAG. 1 Introduction Early Retrieval-Augmented Generation (RAG) took the form of a weighted ensemble: “the top K documents are retrieved using the retriever, and the generator produces the output sequence probability for each document, which are then marginalized” ( Lewis et al. 2020 ) . However, subsequent Transformer-based systems shifted from this ensemble-based formulation to Concatenative RAG (ConcatRAG) in which all documents are sequentially concatenated into a single context. ConcatRAG is now standard, and ensemble-based RAG remains relatively under-studied. Modern RAG workloads increasingly require aggregating evidence from tens of retrieved documents in an effective and interpretable manner. ConcatRAG is not well suited to this regime for two reasons: (1) Poor scaling with context length . ConcatRAG incorporates evidence by concatenating many documents into a single long context. As the context grows, attention cost rises quadratically with the number of tokens and the KV cache grows roughly linearly, making inference slow and memory-hungry. Even if the context fits in memory, LLMs can show an unwelcome sensitivity to the position of documents in the context, leading to relevant information being ‘lost in the middle’ ( Liu et al. 2024 ) ; (2) contribution of individual documents is hard to quantify . ConcatRAG fuses all documents into a single context and does not capture each document’s contribution to generation. As a result, reranking and attribution are typically handled by separate components or post-hoc heuristics ( Li et al. 2024 ; Qi et al. 2024 ) . These issues become more pressing as RAG systems face harder queries, rely on deeper retrieval, and are expected to provide clearer evidence to support their outputs. In this work, we re-visit the Ensemble-based RAG formalism and develop practical RAG methodology that addresses the above limitations. We propose Bayesian Ensemble Retrieval-Augmented Generation (BERAG) and Bayesian Ensemble Fine-Tuning (BEFT) , an inference procedure and an end-to-end training method that performs RAG by ensembling generations conditioned on individual documents, rather than concatenating all documents into one long context. BERAG explicitly models a prior and a posterior over documents and uses the posterior as token-level ensemble weights during generation. This Bayesian view yields several direct benefits: (1) Probabilistic reranking : document relevance is estimated by a learned document prior distribution and further refined as a document posterior distribution given generated tokens. (2) Memory parallelism : documents can be processed in parallel as many shorter contexts, which makes it feasible to condition on large numbers of documents under practical VRAM budgets (e.g., Top-K=50 inference on one A100-80G). Note that by document we refer to a general chunk of text-only or multi-modal content. It could correspond to a complete document, a paragraph, or a page of slide, etc; and (3) Interpretable document contribution : the posterior provides a principled measure of each document’s contribution to the generated tokens. BEFT trains BERAG systems end-to-end with simple supervised fine-tuning loss. While our methodology is general, we evaluate BERAG and BEFT primarily on Knowledge-based Visual Question Answering (KB-VQA), where answer generators are expected to work with an imperfect Top- K retrieval list and where the retriever’s Recall@K has been shown to greatly affect final KB-VQA performance ( Lin et al. 2023 ; Yan and Xie 2024 ; Deng et al. 2025 ) . We show that BERAG and BEFT allow systems to make use of the substantially higher recall rate at large K (e.g., Recall@50) without the performance degradation from long context and “lost-in-the-middle” effect seen in standard ConcatRAG. We also evaluate our systems on Document VQA and Multi-Modal Needle-in-a-Haystack, where “focusing” on the right document during generation is vital. Furthermore, we demonstrate that BERAG is unaffected by document ordering in the retrieval list and therefore immune to the “lost-in-the-middle” effect. We show that the posterior distribution over documents can be used to guide “deflection”, i.e., decline to answer when the retrieval results do not contain relevant information. We show that document posteriors can also accelerate BERAG decoding by pruning low-probability documents, achieving faster decoding than ConcatRAG without loss of performance in answer generation. We summarize our contributions below: • We propose BERAG and BEFT, an inference and end-to-end training framework for ensemble-based RAG. • We show strong performance on E-VQA, Infoseek (Sec. 4.2 and 4.3 ), SlideVQA (Sec. 4.4 ), and MMNeedle (Sec. 4.5 ) compared to recent systems , establishing BERAG as a compelling alternative to standard, concatenative RAG. • We demonstrate BERAG resolves the “lost-in-the-middle” effect (Sec. 4.6 ), enables interpretable deflection when the input context is insufficient for question answering (Sec. 4.7 ), and admits document pruning to achieve faster decoding compared to standard ConcatRAG (Sec. 4.8 ). 2 Related Work Knowledge-based Visual Question Answering (KBVQA) In KBVQA, the system is tasked to answer an image-related question based on knowledge in retrieved documents. E-VQA Mensink et al. 2023 and Infoseek ( Chen et al. 2023 ) are widely used Wikipedia-based KBVQA benchmarks, covering knowledge about animals and plants, as well as facts on sites and entities. Recent progress has been driven primarily by stronger retrieval: PreFLMR ( Lin et al. 2023 ) shows that fine-grained late interaction substantially improves document retrieval for KBVQA, EchoSight ( Yan and Xie 2024 ) introduces a dedicated mult-modal reranker to improve retrieval, and ReflectiVA ( Cocchi et al. 2025 ) adds reflective tokens to the model for deciding whether to retrieve and evaluating passage relevance. Document Visual Question Answering (DocVQA) In DocVQA, the knowledge sources are stand-alone multi-modal documents. We study SlideVQA ( Tanaka et al. 2023 ) which is based on a collection of 20-slide presentation and requires

Read the complete text