A Methodology for Assessing Translation Quality Generated by Sign Language Machine Translators: A Case Study with VLibras

Machine translation tools for sign languages are developed to reduce barriers to access to information for deaf people. However, these technologies have some limitations related to the difficulty of dealing with some specific grammatical aspects of sign languages, which can negatively influence the experience of deaf users. To address this problem, we developed a methodology for assessing the quality of the translation of machine translators from Brazilian Portuguese (PT-BR) to the Brazilian Sign Language (Libras). We apply the evaluation methodology as a study case with Libras, a Brazilian machine translate to Libras. In this experiment, some Brazilian human interpreters evaluated the treatment of some specific grammatical aspects in the application. As a result, we observed significant limitations in the translations performed by this application, indicating the need to improve the treatment of these grammatical aspects. Thus, the proposed methodology can be used by researchers and developers in the field to identify limitations and points of improvement in the process of translating a spoken-to-sign language machine translator.