ML Mob at SemEval-2023 Task 1: Probing CLIP on VisualWord-Sense Disambiguation

Successful word sense disambiguation (WSD) is a fundamental element of natural language understanding. As part of SemEval-2023 Task 1, we investigate WSD in a multimodal setting, where ambiguous words are to be matched with candidate images representing word senses. We compare multiple systems based on pre-trained CLIP models. In our experiments, we find CLIP to have solid zero-shot performance on monolingual and multilingual data. By employing different fine-tuning techniques, we are able to further enhance performance. However, transferring knowledge between data distributions proves to be more challenging.