A Construction method of Multilingual Comparable Corpus in the background of Artificial Intelligence and Internet of Things

Comparable corpus is a critical component in the application of the Artificial intelligence and internet of Things (AfoT). AfoT provides a more extensive data source for corpus, which also presents new requirements and challenges for the construction of comparable corpora in adapting to multilingual application scenarios. To meet the need of it, the comparable corpus plays an essential part of research in language information processing and multilingual application scenarios. However, the multilingual comparable corpus is rare, so there is an urgent need to construct multilingual corpus resources. This paper proposes a method for constructing a multilingual comparable corpus, taking a Chinese-Uighur-Tibetan news corpus as an example, and mapping the different language corpus to a unified language vector space. Then, this paper calculates the similarity between different language news texts and serves as a comparability index to construct comparable relations. Through the decision-making mechanism of minimizing the impossibility, it can candidate a comparable corpus pair of multilingual news which of chapter size to realize the construction of a Chinese-Uighur-Tibetan news comparable corpus (CUTCC). After an evaluation analysis, the results shows that our method is superior in accuracy rate and F value compared to existing method. Finally, multilingual comparable corpus constructed in this study provides valuable data resources support and language service for multilingual situations and AfoT application scenarios.

I. Introduction

Artificial Intelligence and Internet of Things (AIoT) is the product of the integration of two technologies: Artificial Intelligence (AI) and Internet of Things (IoT) [1]. The widespread adoption of IoT devices and the explosive growth of data generated by them provide AI with a vast amount of real-world data. AI technology supports the intelligent analysis and empowers processing of data in IoT systems, what’s more, it provides the objective demand and abundant opportunities for the practical application of by the use of corpus [1]. The close relationship between AI and IoT reveal the common needs of corpus, especially comparable corpus. Corpus, a set of well-sampled and processed electronic texts, is the basic resource that is necessary for linguistics theoretical study and natural language processing, especially language engineering of the applications or smart devices of AIoT [2]. As one of the foundational part, comparable corpus have been widely used in smart devices.