BERT Layer Weighting Comparision with Short Text Classification

BERT [Bidirectional Encoder Representations from Transformers] was the first type of neural model which was used hugely with several downstream tasks specially with the generated embedding representation for text classification. But with the existing literature different layer combination was not tested with concatenation as well as mean averaging techniques to represent the features of a text record for short text classification performances. Moreover, it was not properly evaluated with different machine learning algorithms with short text. Therefore, this research work focused on using seven short text type datasets with BERT embedding to identify the best embedding representation and the best machine learning algorithm. Based on conducted experiment, usage of all layers with concatenation techniques was identified as the best embedding representation and bagging as well as support vector machine was identified as the best machine learning algorithm for short text classification. Performance improvements with bagging was more than 6–8% over the other algorithm and concatenation of all layers gave around 5–6 % performance improvements over the other techniques.