Microsoft’s BiomedBERT has returned to Hugging Face’s trending list, with roughly 870,000 downloads reported for the previous month. Released by Microsoft Research in 2020 under the name PubMedBERT, the roughly 110-million-parameter BERT encoder remains a widely used starting point for biomedical classification, extraction, and retrieval systems.

Why scratch training worked

BiomedBERT tested whether a language model trained entirely on biomedical literature could outperform a general-purpose BERT adapted to the domain. The researchers initialized both the model weights and WordPiece vocabulary from scratch, then pretrained them on biomedical text. Their original paper found consistent gains over contemporary models that began with general-domain pretraining.

The trending checkpoint, BiomedBERT-base-uncased-abstract-fulltext, combines PubMed abstracts with full-text articles from PubMed Central. Its vocabulary was also learned from that corpus, reducing the number of subword fragments required for many biomedical terms. Fewer fragments can make entity boundaries easier to learn during tasks such as gene, chemical, drug, and disease recognition.

Read the benchmark in context

Microsoft evaluated the model on BLURB, the Biomedical Language Understanding and Reasoning Benchmark. BLURB combines named entity recognition, PICO extraction, relation extraction, sentence similarity, document classification, and extractive question answering into an aggregate score.