Towards LLM-Powered Task-Aware Retrieval of Scientific Workflows for Galaxy
Shamse Tasnim Cynthia Banani Roy · 2025
Read Towards LLM-Powered Task-Aware Retrieval of Scientific Workflows for Galaxy by Shamse Tasnim Cynthia Banani Roy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-01757.txt.
Introduction
Shamse Tasnim Cynthia and Banani Roy fix a search problem inside Galaxy, the workflow platform that bioinformaticians use to build and share multi-step analyses. Galaxy hosts hundreds of reusable workflows, but its keyword search misses relevant ones whenever the user's words differ from the workflow's, a common failure for long or vague requests.
Their system retrieves in two stages. Dense vector search with modern embedding models gathers candidates. Then instruction-tuned language models, GPT-4o and Mistral-7B, rerank them by how well they fit the task. To evaluate fairly, the authors built a benchmark of Galaxy workflows labeled with topics from BERTopic and realistic task queries synthesized by language models. The two-stage approach significantly improved top-ranked accuracy and relevance in the first systematic study of Galaxy retrieval, and a prototype already runs inside Galaxy.
Catalog record
| CREATOR | Shamse Tasnim Cynthia Banani Roy |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-01757
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards LLM-Powered Task-Aware Retrieval of Scientific Workflows for Galaxy Shamse Tasnim Cynthia Banani Roy Abstract Scientific Workflow Management Systems (SWfMSs) such as Galaxy have become essential infrastructure in bioinformatics, supporting the design, execution, and sharing of complex multi-step analyses. Despite hosting hundreds of reusable workflows across domains, Galaxy’s current keyword-based retrieval system offers limited support for semantic query interpretation that often fails to surface relevant workflows when exact term matches are absent. To address this gap, we propose a task-aware, two-stage retrieval framework that integrates dense vector search with large language model (LLM)–based reranking. Our system first retrieves candidate workflows using state-of-the-art embedding models, then reranks them using instruction-tuned generative LLMs (GPT-4o, Mistral-7B) based on semantic task alignment. To support robust evaluation, we construct a benchmark dataset of Galaxy workflows annotated with semantic topics via BERTopic and synthesize realistic task-oriented queries using LLMs. We conduct a comprehensive comparison of lexical, dense, and reranking models using standard IR metrics, presenting the first systematic evaluation of retrieval performance in the Galaxy ecosystem. Results show that our approach significantly improves top-k accuracy and relevance, particularly for long or under-specified queries. We further integrate our system as a prototype tool within Galaxy, providing a proof-of-concept for LLM-enhanced workflow search. This work advances the usability and accessibility of scientific workflows, especially for novice users and interdisciplinary researchers. I Introduction Scientific Workflow Management Systems (SWfMSs) such as Galaxy [ 1 ] have become indispensable tools in bioinformatics, enabling researchers to design, execute, and share multi-step analyses through visually composed pipelines. The Galaxy platform, in particular, hosts hundreds of reusable workflows across diverse domains, including RNA-seq, ChIP-seq, metagenomics, and proteomics 1 1 1 https://usegalaxy.org/workflows/list_published . However, as the number of publicly available workflows continues to grow, effectively identifying an appropriate workflow based on a user’s high-level biological task or intent remains a major challenge [ 2 ] . Currently, Galaxy’s retrieval capabilities rely heavily on keyword matching over workflow titles and metadata. These lexical methods suffer from well-known limitations such as vocabulary mismatch and poor support for semantic reasoning, often resulting in suboptimal user experience [ 3 , 4 ] . For example, a user query like ”align single-end reads using HISAT2” may fail to retrieve relevant workflows if exact term matches are not present in the metadata. This gap limits the discoverability and reuse of workflows, especially for novice users or interdisciplinary researchers unfamiliar with precise tool names or formats. To address these limitations, this study investigates a broad range of retrieval models—including lexical, dense vector, and LLM-based reranking approaches to identify the most effective method for task-aware workflow retrieval in Galaxy. Recent advances in neural information retrieval (IR) have introduced dense embedding models that encode queries and documents into semantically meaningful vector spaces [ 5 , 6 ] , making them particularly suitable for the scientific domain where queries are often long, descriptive, and diverse in terminology. This study evaluates these retrieval strategies across general-purpose workflows and curated training workflows, with the goal of improving both the accessibility and relevance of retrieved workflows for Galaxy users. Prior studies have explored various strategies to enhance system usability and improve workflow discoverability through more effective search mechanisms [ 4 , 7 , 8 , 9 , 10 ] . For example, Gu et al. [ 4 ] proposed a proxy-based retrieval approach that selects representative workflows for each query to bridge the gap between textual expressions and underlying semantic intent. Other works, such as those by Liu et al. [ 8 ] and Gu et al. [ 7 ] , focused on matching simple keyword-based queries against workflow attributes like titles and module names. Starlinger et al. [ 10 ] applied TF-IDF based similarity scoring over workflow descriptions to support retrieval. Additionally, Diao et al. [ 9 ] incorporated structural workflow elements, such as service relationships, to enrich retrieval accuracy. However, to the best of our knowledge no prior studies have leveraged large language models (LLMs) to support task-aware natural language retrieval in Galaxy or similar scientific workflow environments. Fig. 1: Overview of our proposed framework To address the limitations of keyword-based search in Galaxy, we propose a workflow retrieval approach that combines dense retrieval with large language model (LLM)–based reranking. The primary contributions of this papers are as follows: • We develop a semantic retrieval pipeline that accepts natural language queries and retrieves candidate workflows using state-of-the-art embedding models. These candidates are then reranked using instruction-tuned generative LLMs, enabling fine-grained relevance scoring aligned with user intent. • We construct a high-quality benchmark dataset of Galaxy workflows annotated with semantic topics via BERTopic [ 11 ] , and generate realistic, task-driven queries using LLMs conditioned on workflow metadata. To the best of our knowledge, this is the first effort to synthesize evaluation-grade natural text queries for scientific workflow retrieval at scale. • We conduct a comprehensive evaluation of lexical, dense, and reranking-based retrieval models using standard information retrieval metrics, presenting the first systematic comparison of workflow retrieval effectiveness within the Galaxy ecosystem. Our results demonstrate that the proposed two-stage approach substantially improves retrieval accuracy and semantic relevance. To further validate its practical utility, we integrate our method as a prototype tool, WorkflowExplorer within the Galaxy platform, offering a proof-of-concept for task-aware, LLM-enhanced workflow discovery. Replication Package is available in our online appendix [ 12 ] . Fig. 2: Schematic diagram of the methodology II Related Works Prior research in scientific workflow retrieval has explored a range of techniques, from keyword-based matching to structure-aware semantic similarity. For example, Shao et al. [ 13 ] developed a keyword-based search engine for workflows collected from Kepler, Taverna, and myGrid. Their system demonstrated strong precision in returning relevant workflows that matched users’ keyword queries, offering a practical baseline for early-stage retrieval tasks. Starlinger et al. [ 10 ] at first used bag of words and TF-IDF models to compute the similarity between the query and workflow description. Later in [ 14 ] , they proposed a more structurally sensitive approach that incorporates the execution order of workflow components. By applying topological sorting and normalization techniques, their method enhanced retrieval accuracy for workflows with well-defined procedural sequences. However, this approach requires users to input a complete or partial workflow structure to initiate a query, which limits its accessibility, specially for novice users who lack prior workflow knowledge. Therefore, semantic technologies have been increasingly leveraged to enhance workflow retrieval by capturing deeper relationships beyond keyword matching. For instance, Zhou et al. [ 15 ] introduced a semantic similarity algorithm that incorporates both hierarchical structure and workflow descriptions to enable more effective clustering and recommendation of scientific workflows. Similarly, Yu et al. [ 16 ] proposed a graph-based embedding approach that represents entire workflows across both transactional and scientific domains by combining structural representations with textual metadata. While these approaches move beyond surface-level lexical matching, they often overlook the full structural richness and task-specific semantics embedded in scientific workflows, which can play a critical role in aligning user intent with workflow functionality. In a similar direction, Bergmann et al. [ 17 ] applied graph representations and an A* search algorithm to compute similarity between semantic workflows. While effective in capturing graph-based structure, their technique