Reconstructing Word Representations from Pre-trained Subword Embeddings

Subword embeddings are integral to transformers such as Generative Pre-trained Transformer (GPT) and Bidirectional Encoder Representations from Transformers (BERT), which are utilized in various natural language processing tasks. A subword is a subset of a word and a word can be decomposed into one or more subwords and characters. One challenge with subword tokenization is determining the optimal tokens for representing a word. This research proposes a refinement to the GPT tokenizer based on an analysis of subword tokens from GPT. The proposed approach reconstructs the word with the objective being minimization and refinement of subword tokens in the reconstruction of the word. The evaluation of the proposed approach is done using popular analogy datasets on semantic and syntactic tasks. This research shows the morphological emphasis on reconstructed word embeddings from pre-trained subword embeddings generated from GPT. The proposed approach reports an overall improvement of approximately ten percent for the refined GPT tokenizer on syntactic tasks.