SNCSE: Contrastive Learning for Unsupervised Sentence Embedding with Soft Negative Samples

Unsupervised sentence embedding aims to obtain the most appropriate embedding for a sentence to reflect its semantics. Contrastive learning has been attracting developing attention. For a sentence, current models utilize diverse data augmentation methods to generate positive samples, while consider other independent sentences as negative samples. Then they adopt InfoNCE loss to pull the embeddings of positive pairs gathered, and push those of negative pairs scattered. Although these models have made great progress, we argue that they may suffer from feature suppression, where the models fail to distinguish and decouple textual similarity and semantic similarity. They may overestimate the semantic similarity of any sentence pairs with similar text regardless of the actual semantic difference between them, and vice versa. Herein, we propose contrastive learning for unsupervised sentence embedding with soft negative samples (SNCSE). Soft negative samples share highly similar text but have surely and apparently different semantics with the original samples. Specifically, we take the negation of original sentences as soft negative samples, and propose BidirectionalMargin Loss (BML) to introduce them into traditional contrastive learning framework. Our experimental results on semantic textual similarity (STS) task show that SNCSE can obtain state-of-the-art performance with different encoders, indicating its strength on unsupervised sentence embedding. Our code and models are released at https:// github.com/Sense-GVT/SNCSE.