A Quantitative Analysis of Quality and Consistency in AI-generated Code

With the recent emergence of generative AI (Artificial intelligence), Large Language Model (LLM) based tools such as ChatGPT have become popular assistants to humans in diverse tasks. ChatGPT has also been widely adopted for solving programming problems and for generating source code in software development. This research investigates both the code quality and the consistency of code quality over iterative prompts in 625 ChatGPT-generated Python code samples in the DevGPT dataset and the corresponding code snippets regenerated by manually prompting ChatGPT. Code samples are measured in terms of seven Halstead complexity metrics. We also assess how consistent they are across code snippets generated by different versions of ChatGPT. It was found that while ChatGPT generates good quality code across iterative prompts, it does generate semi-frequent bugs, similar to how humans do, necessitating code review before integration. These traits also remain consistent across code snippets generated by subsequent releases of ChatGPT. These results suggest using AI-generated source code in software development will not hinder the process.