An estimation method of calorie intake using photograph and depth image of dishes in medical facilities

In medical and welfare facilities, the staff visually checks the acquired amount of foods to manage resident health. To reduce the load of the task of the staff, we propose a method that automatically estimates the amount of these meal intakes using images of the meal on the tray. To estimate the amount of change in the food on the tray, we used color as well as depth images of the meals. Using color images, we identified 10 types of foods using machine learning. Consequently, we obtained a food identification accuracy of 85%. Furthermore, we estimated the amount of change in the food on the tray before and after eating. To estimate the change amount of foods, we classified the depth images of each food into six categories and identified the food amounts based on these six categories using the ResNet50 network. As a result, we achieved a 90% accuracy of categorizing for each food type. Using the estimated values, we also proposed a calculation method of the acquired number of calories. We also confirmed that the proposed calorie estimation method was acceptable for use in medical institutions.