CT Open: An Open-Access, Uncontaminated, Live Platform for the Open Challenge of Clinical Trial Outcome Prediction

Jianyou Wang, Youze Zheng, Longtian Bao, Hanyuan Zhang, Qirui Zheng, Yuhan Chen, Yang Zhang, Matthew Feng, Maxim Khan, Aditya K. Sehgal, Christopher D. Rosin, Ramamohan Paturi, Umber Dube, Leon Bergen · 2026

Read CT Open: An Open-Access, Uncontaminated, Live Platform for the Open Challenge of Clinical Trial Outcome Prediction by Jianyou Wang, Youze Zheng, Longtian Bao, Hanyuan Zhang, Qirui Zheng, Yuhan Chen, Yang Zhang, Matthew Feng, Maxim Khan, Aditya K. Sehgal, Christopher D. Rosin, Ramamohan Paturi, Umber Dube, Leon Bergen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16742.txt.

Introduction

Wang and colleagues introduce CT Open, an open live platform running four trial outcome prediction challenges yearly. Submissions are judged on trials whose outcomes were not public at submission time but became public afterwards. Pinning down whether an outcome was public by a date is hard, since registry postings can lag by years. An automated decontamination pipeline uses iterative LLM-powered web search to find the earliest mention of each outcome. Expert annotations validate the pipeline, and the platform releases a training set plus Winter and Summer 2025 test benchmarks.

Catalog record

CREATORJianyou Wang, Youze Zheng, Longtian Bao, Hanyuan Zhang, Qirui Zheng, Yuhan Chen, Yang Zhang, Matthew Feng, Maxim Khan, Aditya K. Sehgal, Christopher D. Rosin, Ramamohan Paturi, Umber Dube, Leon Bergen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67498
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.16742

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CT Open: An Open-Access, Uncontaminated, Live Platform for the Open Challenge of Clinical Trial Outcome Prediction Jianyou Wang, Youze Zheng, Longtian Bao, Hanyuan Zhang, Qirui Zheng, Yuhan Chen, Yang Zhang, Matthew Feng, Maxim Khan, Aditya K. Sehgal, Christopher D. Rosin, Ramamohan Paturi, Umber Dube, Leon Bergen Abstract Scientists have long sought to accurately predict outcomes of real-world events before they happen. Can AI systems do so more reliably? We study this question through clinical trial outcome prediction, a high-stakes open challenge even for domain experts. We introduce CT Open , an open-access, live platform that will run four challenge every year. Anyone can submit predictions for each challenge. CT Open evaluates those submissions on trials whose outcomes were not yet public at the time of submission but were made public afterwards. Determining if a trial’s outcome is public on the internet before a certain date is surprisingly difficult. Outcomes posted on official registries may lag behind by years, while the first mention may appear in obscure articles. To address this, we propose a novel, fully automated decontamination pipeline that uses iterative LLM-powered web search to identify the earliest mention of trial outcomes. We validate the pipeline’s quality and accuracy by human expert’s annotations. Since CT Open’s pipeline ensures that every evaluated trial had no publicly reported outcome when the prediction was made, it allows participants to use any methodology and any data source. In this paper, we release a training set and two time-stamped test benchmarks, Winter 2025 and Summer 2025. We believe CT Open can serve as a central hub for advancing AI research on forecasting real-world outcomes before they occur, while also informing biomedical research and improving clinical trial design. CT Open Platform is hosted at https://ct-open.net/ 1 Introduction Large language model’s performance is increasingly competitive with humans on a wide range of tasks. This progress raises a natural question: beyond assisting humans on tasks they can already perform, can AI systems make reliable predictions on high-stakes problems whose answers are not yet known? In this paper, we study one such problem: predicting the outcome of a clinical trial before the trial has published results or before it has even begun. Clinical trial outcome prediction is both difficult and consequential. If the problem were easy, clinical development would not remain so uncertain, with many launched trials failing. Accurate prediction matters to many, such as clinicians, patients and pharmaceutical companies. For AI research, the challenge of clinical trial prediction is also compelling because it tests whether LLM-based systems can make useful predictions in a domain with immediate and substantial real-world consequences. To facilitate finding a solution to this challenge, we introduce CT Open , a dynamic platform for clinical trial outcome prediction. See Figure 1 for website design. CT Open hosts four challenges each year: Winter Open, Spring Open, Summer Open, and Fall Open. Each challenge corresponds to a three-month window and a fixed set of clinical trial prediction questions. Participants entering a given challenge must submit their predictions before that challenge’s window begins. After the window ends, CT Open evaluates all submissions and releases a leaderboard the following week. Specifically, from the pool of clinical trials associated with that challenge’s question set, CT Open identifies the subset of trials that had no public results before the start of the challenge window but acquired usable public results during it. CT Open ranks submissions using only questions from this subset and their derived answers from public results, which are then released as a time-stamped benchmark. CT Open is distinguished by three properties. First, it is dynamic . Unlike static benchmarks, CT Open will release new time-stamped benchmarks every three-month via its automated, expert-validated pipelines. This helps to evaluate newly released models, while reducing concerns that performance is driven by memorization or benchmark leakage. Second, it is open-access and open-method . Anyone can submit predictions, and participants are not restricted to a predefined evaluation protocol. All methods are permitted. We view this openness as essential: clinical trial prediction is a genuinely difficult open problem, and it is unlikely that a single fixed methodology will be sufficient. Third, CT Open is designed to be contamination-resistant . Our novel and robust decontamination pipeline identifies trials with no publicly available results prior to each challenge’s window, including preliminary, interim, subgroup, conference, journal article, news, blog, financial-disclosure evidence, etc. Since submissions are made before each challenge’s window, strong performance on CT Open for each challenge is much more plausibly attributable to real predictive ability rather than recall of previously published (partial) results. Figure 1: CT Open Platform Screenshots. It has released leaderboards for Winter 2025 and Summer 2025 benchmarks. It is now accepting submissions for Summer 2026 Open. Furthermore, CT Open includes a training set of questions and two time-stamped test benchmarks, Winter 2025 and Summer 2025, with cutoff dates of 2025-01-31 and 2025-08-31, respectively. By construction, no clinical trial included in Winter 2025 or Summer 2025 had any publicly available results before its corresponding cutoff date. Therefore, these benchmarks are suitable for evaluating Large Language Models (LLMs) whose knowledge cutoff, or preferably whose release date, predates the corresponding cutoff timestamp. In this paper, we present initial evaluations that illustrate the range of methods CT Open can support. We compare prompt-based LLMs, retrieval-augmented LLMs, agentic LLMs, traditional machine learning methods and neural networks, and find that simple ML and NN baselines remain competitive with, and sometimes outperform, LLM-based approaches. This highlights an important gap in current LLM capabilities: despite their strong performance on many existing benchmarks, they still struggle on difficult predictive tasks such as clinical trial outcome forecasting. In addition, we show that Retrieval-Augmented Generation (RAG) can provide useful evidence from similar completed trials, although its gains are limited and inconsistent, while agentic search can uncover valuable external information but remains computationally expensive and yields mixed results. Finally, by analyzing certain LLM’s inconsistent behavior across Winter 2025 and Summer 2025, we show why dynamic, time-stamped, contamination-resistant benchmarks are necessary for fair evaluation. 2 Related Work Clinical trial outcome prediction has been studied under a range of formulations, from machine learning and neural network methods over structured drug-development and registry features to more recent multimodal methods that incorporate richer trial context ( 21 ; 29 ; 10 ; 37 ; 44 ; 14 ) . Clinical trial benchmarks ( 7 ; 11 ) mostly target trial-level result, whereas CT Open consists with fine-grained outcome measure level questions with respect to specific study arms. Retrieval-augmented generation (RAG) can help mitigate the long-context limitations of LLMs by providing only the most useful and relevant information ( 6 ) . It also augments an LLM’s parametric knowledge with retrieved information at inference time, improving factual grounding while enabling access to updated external information ( 20 ) . Subsequent work has expanded this paradigm through iterative, self-reflective, and domain-specialized retrieval strategies ( 2 ; 16 ; 40 ) . This is directly relevant to our setting, where access to trial-relevant evidence such as registry entries, prior publications, and drug- or disease-related context may materially affect outcome prediction. Agentic Systems with Tools and Web Access. A related line of work studies agentic LLM systems that interleave reasoning with external actions

Read the complete text