Electricity Corpus Construction based on Data Mining and Machine Learning algorithm

With the exponential increase of the Internet information, it is more difficult to collect and distinguish the useful information online. Corpus plays an important role in information retrieval and recognition. This paper proposes a corpus construction method based on data mining and machine learning for the energy and power industry. Firstly, the information data of the power industry are obtained by web crawler technology. Then they are stored in a database in structured manner. In order to construct a useful power industry corpus, the Long Short-Term Memory - Conditional Random Fields (LSTM-CRF) model are adopted for word segmentation, the Inverse Document Frequency (IDF) feature and Accessor Variety feature are used for new word discovery. The final electricity corpus consists of words, paragraphs and chapters. Based on the information retrieval and knowledge map, the built electricity corpus is demonstrated that is benefit to energy and power research.