Fantastic Animals and Where to Find Them: Segment Any Marine Animal with Dual SAM

As an important pillar of underwater intelligence, Marine Animal Segmentation (MAS) involves segmenting ani-mals within marine environments. Previous methods don't excel in extracting long-range contextual features and over-look the connectivity between discrete pixels. Recently, Segment Anything Model (SAM) offers a universal frame-workfor general segmentation tasks. Unfortunately, trained with natural images, SAM does not obtain the prior knowl-edge from marine images. In addition, the single-position prompt of SAM is very insufficient for prior guidance. To address these issues, we propose a novel feature learning framework, named Dual-SAM for high-performance MAS. To this end, we first introduce a dual structure with SAM's paradigm to enhance feature learning of marine images. Then, we propose a Multi-level Coupled Prompt (MCP) strategy to instruct comprehensive underwater prior infor-mation, and enhance the multi-level features of SAM's en-coder with adapters. Subsequently, we design a Dilated Fusion Attention Module (DFAM) to progressively inte-grate multi-level features from SAM's encoder. Finally, in-stead of directly predicting the masks of marine animals, we propose a Criss-Cross Connectivity Prediction (C³ P) paradigm to capture the inter-connectivity between discrete pixels. With dual decoders, it generates pseudo-labels and achieves mutual supervision for complementary feature rep-resentations, resulting in considerable improvements over previous techniques. Extensive experiments verify that our proposed method achieves state-of-the-art performances on five widely-used MAS datasets. The code is available at https://github.con1IDrchip61IDual_SAM.

1. Introduction

Underwater ecosystems contain a wide variety of marine life, from microscopic plankton to colossal whales. These ecosystems are crucial roles for the earth's environmental balance. Accurate and efficient Marine Animal Segmentation (MAS) is vital for understanding species' distributions, behaviors, and interactions within the submerged world. However, unlike conventional terrestrial images, underwa-ter images include variable lighting conditions, water tur-bidity, color distortion, and the movement of both cameras and subjects. Traditional segmentation techniques, developed primarily for terrestrial settings, often fall short when applied to the underwater domain. Consequently, methods designed to tackle the unique aspects of the marine environ-ment are urgently required for underwater intelligence. Figure 1.

Our inspirations and advantages. (a) Single-position prompt of SAM. (b) Our multi -level prompt. (c) Mutual supervision for our Dual-SAM's decoders. (d) Our Dual-SAM delivers high performances on multiple datasets.