Nutritional assessment system integrating semantic segmentation and point cloud modeling techniques

Abstract

Abstract Balanced nutrition plays a vital role in preventing chronic diseases. We propose a home-based monitoring system that integrates an Nvidia Jetson AGX Xavier embedded device with an Intel RealSense D435 depth camera mounted vertically above the dining table. The system collects data every minute during meals, capturing point clouds with RGB information. Deep-learning object detection models identify and track multiple food items, ensuring each item is recognized independently. Point clouds are aligned using Random Sample Consensus (RANSAC) and Iterative Closest Point (ICP), while Density-Based Spatial Clustering of Applications with Noise (DBSCAN) distinguishes the table, plate, and food. The system records temporal changes in meal portions and reconstructs food point clouds to estimate volume. Each estimate is linked to the corresponding food category predicted by the semantic segmentation model and combined with nutritional values. All data are uploaded to the cloud, enabling user analysis along with preliminary assessments of total nutrient intake and eating speed. By integrating these components, the system provides accurate records of meal volume and composition, supporting comprehensive evaluations of home-based nutrition over a three-month period.