In the industrial production of steel materials, various complex defects may appear on the steel surface owing to the influence of environmental and other ambient factors. These defects are often accompanied by large amounts of background texture information. Especially, some defects with the low resolution and small size are prone to false alarms and missing detections. Aiming to address the issues of these specific defects, this paper proposes a bidirectional cross-scale feature fusion network combined with non-stridden convolution for steel surface defect detection. First, to improve the model's inference speed and reduce the number of parameters, a simple yet effective convolution (PConv), the core component of FasterNet, is introduced in the feature extraction module instead of the traditional ResNet operator. Second, the bidirectional crossing (BiC) module is embedded to construct a bidirectional cross-scale feature fusion network (BiCCFM), which provides more accurate localization clues to enhance the feature representation on small targets. Finally, combined with non-stridden convolution, the SPD-Conv module is developed to aggregate the detection performance of small targets in low-resolution images. Comprehensive experimental results on the public NEU-DET dataset validate the effectiveness of the embedded modules and the proposed model. Compared with other state-of-the-art methods, the proposed model achieves the best accuracy (74.2% mAP @ 0.5) while maintaining a relatively small number of parameters.