Describing images using fuzzy mutual position matrix and saliency-based ordering of predicates

Describing the content based on bounding boxes of objects located within the image has recently gained popularity thanks to the fast development of object detection algorithms based on deep learning. Such description, however, does not contain any information on the mutual relations between objects that may be crucial to understand the scene as a whole. In the paper, a method is proposed that extracts, from the set of bounding boxes, a scene description in the form of a list of predicates containing consecutive objects' position, referring them to previously described ones. To estimate bounding boxes' relative position, a fuzzy mutual position matrix is proposed. It contains the complete information on the scene composition stored in fuzzy 2-D position descriptors extracted from fuzzified relative bounding box coordinates by a two-stage fuzzy reasoning process. The descriptors of non-zero membership function values are next considered as potential predicates related to the image content. Their list is ordered using the saliency-based criteria to select the most relevant ones, explaining best the scene composition. From the ordered list, the algorithm extracts the final list of predicates. It contains complete and concise information on the composition of objects within the scene. Some examples of the proposed method illustrate the paper.