Finder: A Multimodal AI-Powered Search Framework for Pharmaceutical Data Retrieval

Suyash Mishra, Srikanth Patil, Satyanarayan Pati, Sagar Sahu, Baddu Narendra · 2026

Read Finder: A Multimodal AI-Powered Search Framework for Pharmaceutical Data Retrieval by Suyash Mishra, Srikanth Patil, Satyanarayan Pati, Sagar Sahu, Baddu Narendra (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-15623.txt.

Introduction

Mishra and co-authors describe Finder, a search framework built for pharmaceutical data. Older systems struggle with mixed content and manual curation. Finder unifies search across text, images, audio, and video. It blends sparse word matching with dense semantic vectors. A modular pipeline ingests mixed formats and enriches metadata. The system has processed over 291,400 documents, 31,070 videos, and 1,192 audio files. Content spans 98 languages across regulatory, research, and commercial domains.

Catalog record

CREATORSuyash Mishra, Srikanth Patil, Satyanarayan Pati, Sagar Sahu, Baddu Narendra
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27100
LICENSEhttp://creativecommons.org/publicdomain/zero/1.0/
RIGHTSCC0
SOURCEarXiv

Source: https://arxiv.org/abs/2603.15623

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Finder: A Multimodal AI-Powered Search Framework for Pharmaceutical Data Retrieval Suyash Mishra, Srikanth Patil, Satyanarayan Pati, Sagar Sahu, Baddu Narendra Abstract AI is transforming pharmaceutical search, where traditional systems struggle with multimodal content and manual curation. Finder is a scalable AI-powered framework that unifies retrieval across text, images, audio, and video using hybrid vector search, combining sparse lexical and dense semantic models. Its modular pipeline ingests diverse formats, enriches metadata, and stores content in a vector-native backend. Finder supports reasoning-aware natural language search, improving precision and contextual relevance. The system has processed over 291,400 documents, 31,070 videos, and 1,192 audio files in 98 languages. Techniques like hybrid fusion, chunking, and metadata-aware routing enable intelligent access across regulatory, research, and commercial domains. I Introduction Pharmaceutical companies are overwhelmed by the exponential increase in data, the diversity of modalities, and the complexity of regulations, which are the results of advances in biomedicine and the digitization of compliance workflows. The need for rapid access to unstructured content, clinical reports, filings, training materials, and recordings is the key to agility and innovation [ 1 ] . If we look at traditional keyword-based search systems we can see that they cannot handle multimodal content, which means that they are not able to support some formats like annotated slides, transcribed audio, and visual data. The outcome of it is fragmented access, low quality of metadata, and the necessity of a lot of manual work in the processes such as pharmacovigilance and audits [ 2 ] . AI advancements have greatly improved the capabilities of multimodal understanding, and it can now handle natural language reasoning as well [ 3 ] . Such new abilities help a lot when trying to find information in various formats like graphs, scanned documents, and video annotations. This paper introduces Finder , an AI-powered multimodal search framework tailored for pharmaceutical enterprises. Finder ingests content across formats (PDF, Word, Power- Point, audio, video, and images) and converts it into struc- tured textual representations using a modular pipeline. Com- ponents such as OpenAI Whisper for audio transcription and Qwen2 for video captioning are integrated to support comprehensive content normalization. The processed data is enriched with metadata and stored in a document-aware backend, enabling hybrid search capabilities that combine sparse lexical models (BM42) with dense semantic embed- dings (Mixedbread). These models are fused using reciprocal rank optimization to enhance retrieval performance. Compared to task-specific solutions that are only focused on chatbot assistance [ 4 ] , audio summarization [ 5 ] , or visual parsing [ 6 ] , Finder is a single search layer that comprises of scoring for relevance, document chunking, and modality-aware query interpretation. It allows exact retrieval without the need of keyword expertise and is developed to be combined with entity recognition for better context. Finder makes content management better in pharmaceutical environments through metadata-aware routing, user intent inference, and natural language optimization. It allows the accurate retrieval of clauses, versioned documents, and annotated visuals without the need of explicit keywords. Deployed across enterprise environments, Finder supports use cases like audits, literature mining, training retrieval, and cross-functional discovery. Its scalable architecture enables retrieval-augmented generation (RAG), positioning it for generative search and decision-support in healthcare. This paper: • Presents Finder’s architecture and ingestion pipeline • Describes its hybrid search methodology • Evaluates retrieval performance • Outlines future enhancements including re-ranking and conversational retrieval Finder demonstrates how AI-powered search can unlock organizational knowledge, reduce discovery time, and improve decision-making in regulated domains. II Related Works II-A Vision-Language and Multimodal Models Vision-language models (VLMs) jointly process images and text for tasks such as image captioning and visual question answering. The most famous models are CLIP (OpenAI) [ 7 ] , BLIP (Salesforce) [ 8 ] , and Flamingo (DeepMind) [ 9 ] . GPT-4V [ 10 ] and Google Gemini [ 11 ] are examples of such systems that recently have been developed to extend these faculties to chatbots, so that by a combined image and a language encoder they could receive a visual query and handle it. II-B Lexical vs. Neural Retrieval Lexical methods like BM25 [ 12 ] rank documents on the basis of term frequency–inverse document frequency, which are good with exact matches. Neural retrievers (e.g., BERT, DPR [ 13 ] ) create dense embeddings that are used for semantic search. Models like SPLADE [ 14 ] merge sparse lexical signals with learned semantic features trying to get higher recall. II-C Hybrid Retrieval Hybrid retrieval combines the advantages of both lexical and semantic to improve performance. Score fusion (BM25 + embedding similarity [ 15 ] ), hybrid indexing (dense-sparse vectors via ANN), and query expansion (e.g., Multi-Query, HyDE [ 16 ] ) are three different techniques that could be used to make retrieval better. The trend of domain-specific fine-tuning is still more than prevalent for technical fields. II-D LLM-based and Domain-Specific Search RAG (Retrieval-Augmented Generation) links the outputs of LLM to external sources. QA-RAG [ 17 ] in medical field takes user queries and LLM-generate answers, then searches for regulation documents with matching content. Baidu’s EICopilot [ 18 ] , on the other hand, converts natural queries into graph database queries (Gremlin [ 19 ] ) making structured data retrieval more efficient. II-E Search Systems and Infrastructure The current search stacks are utilizing vector databases and approximate nearest neighbor (ANN) algorithms. One of the main drivers of the dense retrieval ecosystems like Pinecone and Zilliz is the FAISS [ 20 ] which allows efficient GPU/CPU indexing for dense retrieval. Various methods such as HNSW and IVFPQ have made it possible to conduct similarity search on large volumes of data. The use of hybrid search modes and benchmarks (e.g., MS MARCO, Natural Questions) not only assists in measuring the effectiveness but also helps in determining the performance of BM25 and neural rankers. III Methodology This section briefly describes the architecture, intelligent pipelines, and AI-integrated modules that are part of the Finder platform. Enterprise-scale multimodal retrieval capabilities such as ingestion, normalization, vectorization, hybrid search, and real-time ranking over large pharmaceutical corpora are all supported by Finder. Fig. 1: Finder Architecture Diagram III-A System Overview Finder is a modular hybrid search engine that utilizes a combination of lexical and semantic techniques to retrieve information from knowledge-intensive domains. It can take in data in any of the mentioned formats, that is, PDF, PowerPoint, Excel, images, audio, and video, then normalize it, enrich with semantic tags, and store textual as well as vector forms in a scalable backend. Retrieval proceeds by tuning queries with a large language model (Claude 4), automatically inferring the filters, and running a hybrid lexical-semantic search with BM42 and Mixedbread. Weighted fusion of fuzzy matching and normalized BM42 scores is used in the final re-ranking stage to emphasize the contextually relevant results. III-B Data Ingestion and Format Normalization Documents are uploaded to Amazon S3 using a standardized object key schema: • s3://bucket/raw/{filename} for unprocessed inputs • s3://bucket/processed/{filename} for normalized outputs Files that are not in PDF format (for example, DOCX, PPTX) are converted to PDF with the help of PyMuPDF. Metadata such as the author, date of creation, and annotations are also transferred. Feature extraction is done with Docling, a tool created

Read the complete text