The contrast between printed defects and background texture is extremely low, and the defects are concealed within the patterns and colors, most existing detection methods are often unable to achieve promising results. This article proposes PUNet, a robust printed fabric defect network based on UNet. First, we designed the P1 block instead of the original convolutional layer of UNet. The P1 block applies a multiscale dilated convolution, enabling the model with stronger generalization and be more lightweight. Second, we introduced a polarized self-attention (PSA) mechanism, which enhances defect detail extraction and captures sensitive information about defective regions of printed fabrics, thereby improving the model's performance for small-sized defect segmentation. Extensive experimental results demonstrate that, compared to the original UNet, our proposed method significantly reduces parameters and computational costs by 63.7% and 58.5%, respectively, on the printed defect dataset and ceramic tile defect dataset. The mean intersection over union (mIOU) achieved is 80.66% and 85.75%, respectively, on the printed fabric defect dataset (PFDD) and tile surface defect dataset (TSDD). Furthermore, PUNet exhibits good detection performance on various representative industrial defect datasets. Compared to the selected state-of-the-art methods, PUNet outperforms them both qualitatively and quantitatively.