An Improved Model for Segmentation and Recognition of Fine-Grained Activities with Application to Surgical Training Tasks

Automated segmentation and recognition of fine-grained activities is important for enabling new applications in industrial automation, human-robot collaboration, and surgical training. Many existing approaches to activity recognition assume that a video has already been segmented and perform classification using an abstract representation based on spatio-temporal features. While some approaches perform joint activity segmentation and recognition, they typically suffer from a poor modeling of the transitions between actions and a representation that does not incorporate contextual information about the scene. In this paper, we propose a model for action segmentation and recognition that improves upon existing work in two directions. First, we develop a variation of the Skip-Chain Conditional Random Field that captures long-range state transitions between actions by using higher-order temporal relationships. Second, we argue that in constrained environments, where the relevant set of objects is known, it is better to develop features using high-level object relationships that have semantic meaning instead of relying on abstract features. We apply our approach to a set of tasks common for training in robotic surgery: suturing, knot tying, and needle passing, and show that our method increases micro and macro accuracy by 18.46% and 44.13% relative to the state of the art on a widely used robotic surgery dataset.

1. Introduction

In this work we approach the problem of automated segmentation and recognition of actions in constrained environments. We define a fine-grained activity as a sequence of action primitives that take place in a specific environment with a finite set of relevant objects. Some examples with this structure include cooking [19], [10], sports [12], and many robotic manipulation tasks [33]. In this paper we target three robotic surgical training tasks: suturing, knot tying, and needle passing Figure 1.

(Top) Image from the suturing video in the jigsaws dataset. Positions of the robot tools are highlighted in blue and red. Insertion points extracted using a deformable parts model are highlighted in green. (Bottom) The timeline depicting which actions occur when. Each of the 10 colors is a unique action primitive.