Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questions

被引：8
|
作者：
Kabir, Samia [1 ]
Udo-Imeh, David N. [1 ]
Kou, Bonan [1 ]
Zhang, Tianyi [1 ]
机构：
[1] Purdue Univ, W Lafayette, IN 47907 USA
来源：
PROCEEDINGS OF THE 2024 CHI CONFERENCE ON HUMAN FACTORS IN COMPUTING SYTEMS (CHI 2024) | 2024年
关键词：
stack overflow; q&a; large language model; chatgpt; misinformation;
D O I：
10.1145/3613904.3642596
中图分类号：
TP18 [人工智能理论];
学科分类号：
081104 ; 0812 ; 0835 ; 1405 ;
摘要：
Q&A platforms have been crucial for the online help-seeking behavior of programmers. However, the recent popularity of ChatGPT is altering this trend. Despite this popularity, no comprehensive study has been conducted to evaluate the characteristics of ChatGPT's answers to programming questions. To bridge the gap, we conducted the first in-depth analysis of ChatGPT answers to 517 programming questions on Stack Overflow and examined the correctness, consistency, comprehensiveness, and conciseness of ChatGPT answers. Furthermore, we conducted a large-scale linguistic analysis, as well as a user study, to understand the characteristics of ChatGPT answers from linguistic and human aspects. Our analysis shows that 52% of ChatGPT answers contain incorrect information and 77% are verbose. Nonetheless, our user study participants still preferred ChatGPT answers 35% of the time due to their comprehensiveness and well-articulated language style. However, they also overlooked the misinformation in the ChatGPT answers 39% of the time. This implies the need to counter misinformation in ChatGPT answers to programming questions and raise awareness of the risks associated with seemingly correct answers.
引用
收藏
页数：17
相关论文
共 50 条
  • [31] Improving Quality of a Post's Set of Answers in Stack Overflow
    Tavakoli, Mohammadreza
    Izadi, Maliheh
    Heydarnoori, Abbas
    2020 46TH EUROMICRO CONFERENCE ON SOFTWARE ENGINEERING AND ADVANCED APPLICATIONS (SEAA 2020), 2020, : 504 - 512
  • [32] An Observational Study on Flask Web Framework Questions on Stack Overflow (SO)
    Albesher, Luluh
    Alfayez, Reem
    IET SOFTWARE, 2024, 2024
  • [33] An Empirical Study on Continuous Integration Trends, Topics and Challenges in Stack Overflow
    Ouni, Ali
    Saidani, Islem
    Alomar, Eman
    Mkaouer, Mohamed Wiem
    27TH INTERNATIONAL CONFERENCE ON EVALUATION AND ASSESSMENT IN SOFTWARE ENGINEERING, EASE 2023, 2023, : 141 - 151
  • [34] Does This Apply to Me? An Empirical Study of Technical Context in Stack Overflow
    Galappaththi, Akalanka
    Nadi, Sarah
    Treude, Christoph
    2022 MINING SOFTWARE REPOSITORIES CONFERENCE (MSR 2022), 2022, : 23 - 34
  • [35] An empirical study of IoT topics in IoT developer discussions on Stack Overflow
    Uddin, Gias
    Sabir, Fatima
    Gueheneuc, Yann-Gael
    Alam, Omar
    Khomh, Foutse
    EMPIRICAL SOFTWARE ENGINEERING, 2021, 26 (06)
  • [36] An empirical study of IoT topics in IoT developer discussions on Stack Overflow
    Gias Uddin
    Fatima Sabir
    Yann-Gaël Guéhéneuc
    Omar Alam
    Foutse Khomh
    Empirical Software Engineering, 2021, 26
  • [37] The reproducibility of programming-related issues in Stack Overflow questions
    Saikat Mondal
    Mohammad Masudur Rahman
    Chanchal K. Roy
    Kevin Schneider
    Empirical Software Engineering, 2022, 27
  • [38] Predicting Tags of Stack Overflow Questions: A Deep Learning Approach
    Subramani, Srinivas
    Rajesh, Sangeetha
    Wankhede, Kirti
    Wukkadada, Bharati
    2023 Somaiya International Conference on Technology and Information Management, SICTIM 2023, 2023, : 64 - 66
  • [39] Seahawk: Stack Overflow in the IDE
    Ponzanelli, Luca
    Bacchelli, Alberto
    Lanza, Michele
    PROCEEDINGS OF THE 35TH INTERNATIONAL CONFERENCE ON SOFTWARE ENGINEERING (ICSE 2013), 2013, : 1295 - 1298
  • [40] Community evolution on Stack Overflow
    Moutidis, Iraklis
    Williams, Hywel T. P.
    PLOS ONE, 2021, 16 (06):
12345