Estimation of the Caloric Intake of Food Consumption Using Convolutional Neural Network

Dietary assessment is commonly used to monitor the calorie intake of people who wish to regulate their diet. Traditional dietary assessments require participants to actively record their meals for 24 hours, followed by a manual assessment of calorie intake by a health professional. This approach is not effective for use on a daily basis, and thus cannot give a comprehensive analysis of the participant’s diet plan. New dietary assessment techniques are needed to make food monitoring easier and more feasible in a real-world environment. Many research works use image-assisted or image-based techniques to simplify the process and make it more acceptable to the participants. Yet, these works generally focus on analyzing the main course of a meal, and the meal is usually set up in a laboratory setting with a simple background. While finger food plays an important role in diet, as it is easy to lose track of the amount consumed, it is a challenge for dietary assessment. As a result, estimating the total calorie intake of the participant is difficult. In this paper, we present a real-time food monitoring technique that uses computer vision techniques to passively detect when finger food is being grabbed by participants during a live stream, to classify the amount of potato chips grabbed and give immediate feedback to the participants. Two convolutional neural networks (CNN) are used for this purpose: one for detecting the participants’ hands and the other for classifying the chips according to calorie labels. In order to train the CNN to estimate calorie intake, we collected a dataset with more than 26,600 images of volunteers grabbing differing quantities of chips. The macro-average F1 score of our models ranges from 0.75 to 0.88 for a 3-class system, and it ranges from 0.59 to 0.74 for a 4-class system.