Controllable Text Generation Based on Enhanced Non-Residual Attention

The typical construction method of prompts in CLM results in the combination of prompt text information and input text information being too long, and designing a prompt is challenging. Fixed template combinations can impact the grammatical structure, complicate the understanding of the prompt model, and influence the control effect of the generative model. By enhancing Non-Residual Attention, the prompt model processes prompt information to generate improved prompts. This enables the generative model to access comprehensive enhanced prompt information and input text information at any time step. At the same time, a copying mechanism is incorporated into the generative model to tackle the consistency issue of contextual text and enhance the controllability of the generative model. This enables the model output to encompass input text information or prompt text information, thereby reflecting contextual consistency and improving output control. Based on the ROCStory dataset with labeled characters, emotions, and actions, the results indicate that the enhanced Non-Residual Attention model has better control and generation effects compared to the original method.

I. Introduction

Controlled Text Generation (CTG) refers to the use of machine learning techniques to generate natural language text while controlling the features and content of the text, such as language style, theme, emotion, and more. This technology has wide-ranging uses across areas like machine translation, automated dialogue systems, automated summarization, and sentiment analysis. The background of controllable text generation stems from the fact that humans have a strong control over language use, which allows them to use different language styles and control the expression of emotions and other characteristics in different contexts. Therefore, enabling machines to perform controllable text generation can better fulfill people's needs. In practical applications, controllable text generation technology can help businesses quickly create text content such as business reports, marketing copy, and news reports, thereby reducing labor and time costs. In addition, controllable text generation can also be used to create personalized dialogue systems, thereby enhancing user experience and interactive effects. Controllable text generation proves crucial to promoting social and economic development, as evidenced by these application backgrounds.