Fabric defect detection can greatly enhance the quality of fabric production. However, the high cost of annotating defects and the computational complexity of networks are the main challenges in defect detection. To address these challenges, this paper proposes a semi-supervised lightweight fabric defect detection algorithm (SDA-Net). During the semi-supervised training process, the algorithm uses labeled defect samples and normal samples to learn latent features and detect defect positions accurately. First, to solve the issue of insufficient labeled defect samples due to high annotation costs, a data augmentation method called Sel-fill is proposed. The Sel-fill randomly samples image blocks of various sizes from a truncated normal distribution. These image blocks are then inserted into random positions within normal images, thereby generating labeled defect samples. Second, A lightweight neural network architecture is constructed using depth-wise separable convolution (DSConv). This architecture effectively reduces the number of parameters and computations while maintaining performance. Final, the max pooling coordinate attention mechanism (MpCA) effectively suppresses background noise during the multi-scale feature fusion process, resulting in improved detection precision. By using depth-wise separable convolution and MpCA attention, SDA-Net achieves an average detection precision of 62.6%, improved by 4.5% over the previous method. The number of trainable parameters is only 9.35 MB, reduced by 42.53%. Moreover, the computations are reduced by 68.84%.