An Optimisation Study of Attention Mechanism for Intent Recognition of Short Texts in Transformer Framework

In this paper, we investigate the use of attention mechanism for short text intent recognition under the Transformer framework. The approach combines deep learning techniques with the introduction of a pointer generation network to optimise the decoder part of the Transformer model, where both generation probability and replication probability are computed at each time step of the decoder, where the generation probability is used to generate a new word and the replication probability is used to select a word from the input sequence. Thereby, at each time step, the model can select the most appropriate word as an output based on the context and the generation/copy probability. By optimising the decoder, the performance of the model in handling the task of short text intent recognition is improved. Experimental results show the effectiveness and superiority of the method in the short text categorisation intent recognition pathway. The accuracy and performance of short text intent recognition can be improved on datasets such as SNIPS.