Camera-based Progress Estimation of Assembly Work Using Deep Metric Learning

In this paper, a progress estimation method using deep learning is proposed to visualize the product assembly process in a factory. First, the target assembly product is detected from images acquired from a fixed-point camera installed in the factory using a deep learning-based object detection method. Next, the detection area is cropped from the image. Finally, by using a classification method based on deep metric learning on the cropped image, the progress of the product assembly work is estimated as a rough progress step. In addition, considering the similarity of features with neighboring steps when learning with deep metric learning, we propose an adaptive loss function that learns to separate features from nearby steps. In experiments, an 82 [%] success rate is achieved for the progress estimation method using deep metric learning. Furthermore, the method using the adaptive loss function achieved a success rate of 92 [%]. Experiments were also conducted to verify the practicality of a series of detection, cropping and progress estimation.