Nutrient Intake Estimation System Integrating Semantic Segmentation and Point Cloud Modeling Techniques

The object recognition model within the deep learning technology was used to extract food positions, generating point cloud files containing RGB information that represented the gradual reduction of dishes at different times and positions. For data collection, the depth camera and edge devices were used to acquire point cloud data. Simultaneously, semantic segmentation techniques were employed for content analysis to calculate corresponding nutritional values. We used edge devices (Nvidia Jetson TX2) and depth cameras (Intel RealSense D435) and employed various advanced techniques, including deep learning, object recognition, semantic segmentation, and point cloud modeling. The handling of point cloud data was pivotal in this research. Random Sample Consensus (RANSAC) and Iterative Closest Point (ICP) algorithms were used to facilitate rapid pairing and calibration of point clouds. Through cluster analysis (DBSCAN), coordinates of desktops, plates, and food in the point cloud were distinguished, isolating the point cloud data of food and progressively reconstructing its real-world appearance layer by layer. Finally, volume estimation was performed for reconstructed food entities and underlying container surfaces provided by the point cloud. Then, the point cloud into triangular meshes was transformed to extract vertex coordinates for triangular volume computation, thereby calculating the food volume. The application of such technologies allowed the system to accurately record the volume and composition of food. Additionally, it enabled the analysis of dietary behaviors, thereby providing significant support for intelligent home care.