Pose Estimation-Based Visual Perception System for Analyzing Fish Swimming

Advances in modern deep learning-based computer vision perception techniques have revolutionized animal movement research methods. These techniques have also opened up new avenues for studying fish swimming. To that end, we have developed a visual perception system based on pose estimation to analyze fish swimming. Our system can quantify fish motion by 3-D fish pose estimation and dynamically visualize the motion data of marked keypoints. Our experimental results show that our system can accurately extract the motion characteristics of fish swimming, which analyze how fish bodies and fins work together during different swimming states. This research provides an innovative idea for studying fish swimming, which can be valuable in designing, developing, and optimizing modern underwater robots, especially multifin codriven bionic robotic fish. The code and dataset are available at https://github.com/wux024/AdamPosePlug.

I. Introduction

Advanced deep learning-based computer vision perception techniques have facilitated the rapid development of animal behavior analysis tools that have greatly enhanced the ability of neuroscience and zoology scientists to conduct relevant research [1], [2], [3]. Among the behaviors of many species, we are more interested in the swimming of fishes because studying the swimming of fishes may reveal inspiring knowledge valuable for developing new types of robotic fishes [4].