Analogy-based Assessment of Domain-specific Word Embeddings

The ability of word embeddings to identify shared semantic regularities between word pair categories such as capital-country has led to the use of analogies as a method of validating word embedding models. Further research has shown that relative to the complete breadth of possible analogy categories, there exists a limit to the particular categories accessible, in terms of accuracy, to current analogy equations executed against word embeddings trained on generalized, non domain-specific text corpora. As most, if not all, domain-specific, scientific analogy pairs belong to problematic analogy categories (i.e. the lexicographical and the encyclopedic), we examine the degree to which a domain-specific text corpus and vocabulary positively improve analogy predictions from word embeddings. Our findings demonstrate that in comparison to analogy-based tests performed against general word embeddings, predictions by domain-specific word embeddings outperform in exactly those analogy categories that are both highly problematic and the location of domain knowledge.

I. Introduction

Analogies are recognized as a method for validating word embeddings [16] [22]. Typically, both the word embeddings and the analogy test sets are built from generalized text corpora and generalized vocabularies. Recent research has examined the performance of word embeddings built from domain-specific text corpora and trained using domain-specific vocabularies [7]. Our research tests the hypothesis that word embeddings built from a domain-specific, Earth science corpus and trained using domain-specific vocabulary will better predict domain-specific, Earth science analogies when compared with the results achieved by tests of non domain-specific analogies against word embeddings produced by generalized corpora. Further, we tested the hypothesis that the improvement in predictions would occur in the categories of analogical relationships in which most, if not all, domain knowledge is to be found.