PSCSE: prompt-based contrastive learning with sample filtering for unsupervised sentence embedding

Abstract

Abstract Unsupervised sentence representation learning is critical in natural language processing. Recently, contrastive learning methods achieved remarkable performance by optimizing the alignment and uniformity of embedding spaces. Nevertheless, the dominant methods mainly focus on the data augmentation for the positive samples but ignore the sampling approach for the negative samples. Most methods rely on the random in-batch sampling approach for the generation of the negative samples. This approach could result in false negatives and create sampling biases for the model’s discriminative ability. To address this problem, we propose a new method prompt-based contrastive learning with sample filtering for unsupervised sentence embedding (PSCSE). In the proposed model, we used the synthesized hard negatives generated by the “NOT”-style prompts (e.g., “This sentence: [X] does not mean [MASK]”) to optimize the uniformity of the learned representations. In addition, we used the auxiliary encoder for the sample filtering approach to address the false negatives. We conducted experiments on the semantic textual similarity dataset and achieved remarkable performance by surpassing the dominant methods SimCSE, E-SimCSE, and PromptBERT by up to 1.0 points in the average Spearman’s correlation score.