Benchmarking framework for anomaly localization: Towards real-world deployment of automated visual inspection

Localizing defects in products is a critical component of industrial pipelines in manufacturing, retail, and many other industries to ensure consistent delivery of high quality products. Automated anomaly localization systems leveraging computer vision have the potential to replace laborious and subjective manual inspection of products. Recently, there have been tremendous efforts in this research domain investigating deep learning-based anomaly localization methods. However, such proposed methods, mainly considering product-specific evaluation, cannot be directly implemented for real-world use. Therefore, despite the advancements, there is still a gap between research and deployment of those methods to real-world production environments. Implementing any automated solution for manufacturing can involve a steep upfront investment. It is important to develop an industry-friendly benchmarking framework to ensure the feasibility and robustness of an automated quality control system. In this paper, we present a new anomaly localization benchmarking framework considering different aspects -(1) understand the performance of models in a generalizable product-agnostic manner, (2) explore pros and cons to find the most optimal modeling approach, (3) develop an efficient training and inference scheme with defect-free training samples and very few defective samples for evaluation, and (4) perform an ablation study of threshold estimation techniques to determine optimal threshold level for segmentation. We release a newly-labeled dataset for the research community with product-agnostic categorization of defective product images. To the best of our knowledge, this is the first anomaly localization work on developing a benchmarking framework focusing on real-world use. We believe domain experts from different industries will find this useful and can gain valuable insights to deploy automated visual inspection in production pipelines.