TCS: A Teacher-Curriculum-Student Learning Framework for Cross-Lingual Text Labeling

In recent years, deep learning models have greatly promoted the development of English and Chinese natural language processing. However, most other languages in the world are unable to perform effective text processing and analysis because of the difficulty in obtaining labeled data. Cross-lingual transfer is the main way to solve this problem by using labeled samples in the source language to make the model learn the corresponding tasks in the target language, so it has been widely concerned. Recently, some works based on self-training achieve the best results in the cross-lingual text labeling tasks by using both labeled samples in the source language and unlabeled samples in the target language to fine-tune the multilingual BERT. However, self-training suffers from the problem of inaccurate supervision, that is, the teacher model's inaccurate predictions on the target unlabeled samples (i.e. inaccurate samples) may mislead the subsequent student model. And in cross-lingual transfer scenarios, the natural distribution gap between the source labeled samples and the target unlabeled samples will make this problem even worse. In order to further improve the results of cross-lingual text labeling, in this paper, we utilize three techniques to address the inaccurate supervision problem in the self-training, and propose a learning framework called Teacher-Curriculum-Student (TCS). Firstly, we employ a soft-target training technique to reduce the impact of inaccurate samples at the level of loss function. Secondly, we employ a progressive sample selection technique to construct iterative training datasets containing more accurate samples. Finally, in order to handle the inaccurate samples in iterative training datasets, we propose a from-confident-to-suspicious curriculum learning technique: according to prediction confidences of the teacher model, a training dataset is organized into learning courses arranged from confident to suspicious, so as to enhance the role of accurate samples and reduce the role of inaccurate samples in the student model's training process. Experiments on the benchmark datasets of cross-lingual text classification and cross-lingual named entity recognition show that the average results obtained by TCS are improved by 2.51% and 3.25% respectively on the basis of self-training thanks to the three techniques TCS used, and are 1.51% and 4.45% higher than existing state-of-the-art results respectively. In addition, ablation experiments show that: all three techniques used in TCS can effectively improve the performance of the final model and the curriculum learning technique contributes to the largest increase; the from-confident-to-suspicious curriculum order is the key to the effectiveness of the curriculum learning technique in the self-training scenario. More interestingly, further analysis shows that in the whole iterative training process, the effect of TCS is always better than that of self-training, the sample selection technique plays a role in the initial iteration, and the effect of the curriculum learning technique is mainly reflected in the middle and later iteration. For reproducibility, we release the code and experimental configurations at https://github.com/ericput/TCS. © 2022, Science Press. All right reserved.