Automatic segmentation of clustered breast cancer cells using watershed and concave vertex graph

Automatic segmentation of stained breast tissue images helps pathologists to discover the cancer disease earlier. Separation of touching cells presents many difficulties to the traditional segmentation algorithms. In this paper, we propose a new automatic method to segment clustered cancer cells. In the first step, we detect cell regions using a modified geometric active contour based on Chan-Vese energy functional. Then, touching cell regions are extracted from the pre-segmented image by detecting high concavity points along the cell contours. A gradient-weighted distance transform is used in the watershed algorithm in order to get the most significant inner edges. To solve the problem of over-segmentation, which is the major drawback of the watershed method, a combination of three techniques is presented as a post-processing step. First, the nearest end points to concave vertices are detected in the inner edges in order to get the initial separating curve candidates. Second, a concave vertex graph is constructed from the end points and the separating curves. Finally, Dijkstra algorithm is applied to find the shortest path that separates the touching cells. The proposed algorithm is tested on several breast cancer cell images and it's compared with the classical watershed algorithm and a recent marker-controlled watershed method. The experimental results show the performance of the presented approach.