Automatic detection of defects on painted wall surfaces (DPWSs) based on machine vision is meaningful for reducing manpower consumption and shorting lead time, which is one of the critical components of intelligent construction. Conventional detection methods suffer from some challenges due to the multi-scale defects and unstructured detection environment. In this study, a detection network for DPWSs is developed based on the enhanced You Only Look Once version 5 (YOLOv5). First, the convolutional block attention module (CBAM) is inserted into the backbone of YOLOv5 to boost the feature extraction and suppress noise, which can sufficiently extract the features of the defects with blurry edges. Then, to improve the adaptability for multi-scale defects and reduce the model size, the Bi-directional Feature Pyramid Network (BiFPN) is employed in the neck of YOLOv5 to enhance the feature fusion, where the multi-scale objects can be fully captured. Finally, the decoupled head is proposed to replace the original convolution layer in the You Only Look Once (YOLO) head, which separates the classification and localization tasks to improve detection speed and robustness. Since there is no publicly available data set, a data set of DPWSs is constructed, and a series of comparative experiments are conducted. The results show that the detection accuracy is improved by 15.6% and the model size is reduced by 30.8% compared with YOLOv5. Meanwhile, the proposed network has better adaptability to DPWSs with higher detection accuracy and smaller model sizes compared with other advanced methods. As to the general applicability aspect of the model, the proposed model holds significant academic and practical implications in the realms of intelligent construction. Besides the model’s primary application domain of construction quality control, it can also be applied to defect detection in other scenarios that have multi-scale defects and unstructured environments. This versatility benefits a wide spectrum of construction projects. © The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature 2024.