A Real-Time Background Replacement Method Based on Machine Learning for AR Applications

Recent technological advances in Virtual Reality (VR) and Augmented Reality (AR) enable users to experience a high-quality virtual world. Using VR to experience the virtual world, the user's entire view becomes the virtual world, and the user's physical movement is generally limited because the user cannot see the surrounding situation in the real world. Using AR to experience the virtual world, we generally use special sensors such as LiDAR to detect the real space and superimpose the virtual world on the real space. However, it is difficult for devices without such special sensors to detect real space and superimpose a virtual world at an appropriate position. This study proposes two methods for replacing the background: a method using depth estimation and a method using semantic segmentation. This study also confirmed that the system can be used with sufficient removal accuracy and response time by using appropriate image size for the environment and that a safe and highly immersive virtual world experience can be achieved.