Constructing a Relevance-oriented Dataset for Training Transformer Rankers for Medical Search

As there is usually sparse relevance assessment data for training, state-of-the-art transformer-based rankers may under-perform when directly employed in the medical domain. In particular, the lack of human-annotated labels has become a bottleneck of learning the data-hungry deep IR models. In this work, we introduce an effective approach to generating data from existing medical corpora for training transformer-based re-ranking models. Inspired by the fact that the title is usually closely relevant to the abstract in a scientific article, we create a dataset by regarding titles as queries and the corresponding abstracts as relevant documents, and use an unsupervised IR model (e.g., BM25) to sample negative examples. Thereafter, we train the re-ranking model on this relevance-oriented dataset before training the model on the target collection. Experimental results on the standard TREC Precision Medicine track benchmark show that our approach, despite its simplicity, is able to outperform strong recent baselines. The dataset is available for download
¹
.