Film Video and Background Music Retrieval Based on Emotion

Audio-visual multimodal information has become the focus of research in various multimedia fields, among which the cross-modal research of video and audio is an important direction of current research. In this paper, a new film video and background music cross-modal retrieval network was established based on emotion. It includes an audio sub-network and a video sub-network. The audio sub-network is used to extract audio features from the audio input of the audio-visual pairs, while the video sub-network is used to extract video features. The features extracted from different modalities are then measured for their similarity in a shared space, and optimized and updated through a loss function. In order to make better use of the emotional information of audio and video, a fusion loss function combining the discrimination loss and retrieval loss within the mode is adopted. The experimental results show that the proposed method is effective from subjective and objective evaluation.