DSSF: Dynamic Semantic Sampling and Fusion for One-Stage Human–Object Interaction Detection

Human–object interaction (HOI) detection is a fundamental task for machines to understand high-level scenarios. Nowadays, most conventional HOI detectors are usually computationally inefficient due to the decoupled two-stage manner. Although the limitation of cascaded object detection is broken through by the latest one-stage methods of direct parallel detection for HOI triplets, the problem of insufficient extracted features and lack of HOI specific interactive semantic context still exists. Therefore, an improved one-stage framework is proposed in this article, in which dynamic semantic sampling and fusion (DSSF) play key roles. Given the observation that numerous HOI classes have very limited positive samples, we tackle the long-tailed challenge by sampling dynamically using interaction semantic regularities. Meanwhile, the semantic expression ability of interaction points is explicitly enhanced by the dynamic semantic fusion (DSF) with aggregated contextual information. Moreover, we design a feature fusion module for parallel branches to reduce the conflict of multitask optimization, along with a point matching strategy (PMS) to filter out low-probability HOI pairs. Finally, without introducing extra features, our DSSF outperforms previous state-of-the-art methods by a large scale on two challenging HOI detection benchmarks: V-COCO and HICO-DET.