The field of computer vision has experienced rapid progress owing to deep learning. The importance of road damage detection in ensuring traffic safety and reducing road maintenance costs is becoming increasingly evident. For detecting road damage, the YOLOv5 algorithm provides a reliable and effective method. However, YOLOv5 still requires a significant amount of computation. This paper proposes a lightweight network for detecting road damage that improves upon the YOLOv5 model in four ways. The algorithm accurately identifies and classifies different types of road damage, while simultaneously reducing the number of parameters and required computations. First, lightweight processing of the model is achieved. The Ghost module and Ghost Bottleneck are employed to construct the novel GBS module and C3Ghost, which replace the existing CBS and C3 modules. Second, the CIoU loss function is transformed into SIoU to improve the precision of target box regression. Furthermore, the original upsampling module is replaced by CARAFE to improve the model's semantic adaptability and receptive field. Finally, the CBAM attention mechanism is employed to concentrate on crucial feature information. The experiment's findings present that, in comparison to the baseline model, the upgraded model has 41.8% fewer parameters. Additionally, there has been a 43.8% reduction in floating-point computation and an improvement of 0.2% in detection accuracy.