Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction.
Rasmy L, Xiang Y, Xie Z, Tao C, Zhi D. · 2021
Read Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. by Rasmy L, Xiang Y, Xie Z, Tao C, Zhi D. (2021) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc8137882.txt.
Description
Rasmy and co-authors pretrain Med-BERT, a transformer model that reads structured health records the way BERT reads text. Hospital records encode diagnoses and visits as sequences of codes. Deep models usually need huge local cohorts to learn from them. Most clinics cannot assemble such cohorts alone. Med-BERT learns general record patterns first. Pretraining covered records from 28,490,650 patients in one structured dataset.
Fine-tuning then adapts the model to a single disease task. Two prediction tasks on two databases tested the transfer. Pretraining lifted accuracy in both, the authors report. The gain shone brightest with tiny training sets. Small-data models gained more than 20% in score by their measure. One matched a rival trained on ten times the data. Shared pretraining could spare clinics costly data hunts. Rare-disease prediction stands to gain the most. The record's grammar, once learned, travels between hospitals.
Catalog record
| CREATOR | Rasmy L, Xiang Y, Xie Z, Tao C, Zhi D. |
|---|---|
| YEAR | 2021 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| LICENSE | CC BY 4.0 |
| RIGHTS | CC BY |
| SOURCE | Europe PMC |