Bridging the Gap between Semantics and Multimedia Processing

In this paper, we give an overview of the semantic gap problem in multimedia and discuss how machine learning and symbolic AI can be combined to narrow this gap. We describe the gap in terms of a classical architecture for multimedia processing and discuss a structured approach to bridge it. This approach combines machine learning (for mapping signals to objects) and symbolic AI (for linking objects to meanings). Our main goal is to raise awareness and discuss the challenges involved in this structured approach to multimedia understanding, especially in the view of the latest developments in machine learning and symbolic AI.

I. Introduction

A classic problem in multimedia representation and understanding is the semantic gap problem [1]. It states that there is a big representational gap between the audiovisual signals that compose multimedia objects and the concepts represented by these signals. For instance, the dominant color and movement trajectory of a given set of pixels in a video clip, which are low-level characteristics of the clip, usually do not provide much information about the meaning of the set of pixels—at least not to computers. But recent developments in artificial intelligence (AI) are changing that.