Determining the Amount of Information in One Information Bit of Text Data

With the rapid growth of the amount of transmitted information in the network and its continuous increase, the need to optimize not only technologies and hardware and software for data transmission, but also to develop an algorithm for preprocessing information for its value to the end user inevitably increases. This article is devoted to the development of such an algorithm. To do this, the essences of natural language processing will be considered, a study will be conducted and the stages of preparing text data for recognition by a machine will be determined, and calculations performed on the text will be carried out, allowing an extensive analysis of text data for its value and identification of patterns in them.