Variable-order N-gram generation by word-class splitting and consecutive word grouping

In this paper, a generation scheme for variable-order N-grams is proposed to attain reliable statistical constraints from a given language corpus. Starting from POS bigrams, the proposed scheme creates variable-order N-grams by splitting a POS into finer groups and by adding frequent consecutive word sequences as word-classes. This word-class splitting and consecutive word grouping are carried out incrementally by minimizing the total entropy. Experiments showed that the perplexity of the proposed model for the test corpus is lower than that for a conventional trigram and that this model requires a quite smaller number of statistical parameters. By applying this model to speech recognition, we get a better recognition rate than using conventional bigrams.