A commentary of GPT-3 in MIT Technology Review 2021

Through the development of large-scale natural language models with writing and dialogue capabilities, artificial intelligence (AI) has taken a significant stride towards better natural language understanding (NLU) and human-computer interaction (HCI). As of today, the GPT-3 model, developed by OpenAI, is the language model with the most parameters, the largest scale, and the strongest capabilities. Using a large amount of Internet text data and thousands of books for model training, GPT-3 can imitate the natural language patterns of humans nearly perfectly. This language model is extremely realistic and is considered the most impressive model as of today. Despite its powerful modeling and description capabilities, there are significant issues and limitations. First and foremost, the GPT-3 model does not understand writing (natural language generation) well and sometimes generates uncontrollable content. Secondly, training the GPT-3 model requires a large amount of computing power, data, and capital investment, and releases significant carbon dioxide emissions. Developing similar models is only possible in laboratories with adequate resources. Furthermore, as the GPT-3 model is trained with Internet text data rife with error messages and prejudices, it often produces chapters and paragraphs with biased content similar to the training data.