On the Reliability of User Feedback for Evaluating the Quality of Conversational Agents

被引：0
|
作者：
Massiah, Jordan [1 ,2 ]
Yilmaz, Emine [1 ,2 ]
Jiao, Yunlong [1 ]
Kazai, Gabriella [1 ]
机构：
[1] Amazon, London, England
[2] UCL, London, England
来源：
PROCEEDINGS OF THE 32ND ACM INTERNATIONAL CONFERENCE ON INFORMATION AND KNOWLEDGE MANAGEMENT, CIKM 2023 | 2023年
关键词：
conversational agent quality; user feedback reliability;
D O I：
10.1145/3583780.3615286
中图分类号：
TP18 [人工智能理论];
学科分类号：
081104 ; 0812 ; 0835 ; 1405 ;
摘要：
We analyse the reliability of users' explicit feedback for evaluating the quality of conversational agents. Using data from a commercial conversational system, we analyse how user feedback compares with human annotations; how well it aligns with implicit user satisfaction signals, such as retention; and how much user feedback is needed to reliably evaluate the quality of a conversational system.
引用
收藏
页码：4185 / 4189
页数：5
相关论文
共 50 条
  • [21] Expressing Personalities of Conversational Agents through Visual and Verbal Feedback
    Lee, Seo-young
    Lee, Gyuho
    Kim, Soomin
    Lee, Joonhwan
    ELECTRONICS, 2019, 8 (07)
  • [22] Design and Implementation of Conversational Agents for Harvesting Feedback in eLearning Systems
    Lundqvist, Karsten O.
    Pursey, Guy
    Williams, Shirley
    SCALING UP LEARNING FOR SUSTAINED IMPACT, 2013, 8095 : 617 - 618
  • [23] Evaluating the quality and dissemination of teacher-developed inquiry materials: Analysis of user feedback
    Carmel, Justin H.
    Yezierski, Ellen J.
    Herrington, Deborah G.
    ABSTRACTS OF PAPERS OF THE AMERICAN CHEMICAL SOCIETY, 2011, 241
  • [24] Methods for Evaluating Conversational Agents' Communicability, Acceptability and Accessibility Degree
    Valtolina, Stefano
    Matamoros, Ricardo Anibal
    Epifania, Francesco
    HUMAN-COMPUTER INTERACTION - INTERACT 2023, PT II, 2023, 14143 : 372 - 382
  • [25] Evaluating the Effect of Gesture and Language on Personality Perception in Conversational Agents
    Neff, Michael
    Wang, Yingying
    Abbott, Rob
    Walker, Marilyn
    INTELLIGENT VIRTUAL AGENTS, IVA 2010, 2010, 6356 : 222 - 235
  • [26] User-Regulation Deconfounded Conversational Recommender System with Bandit Feedback
    Xia, Yu
    Wu, Junda
    Yu, Tong
    Kim, Sungchul
    Rossi, Ryan A.
    Li, Shuai
    PROCEEDINGS OF THE 29TH ACM SIGKDD CONFERENCE ON KNOWLEDGE DISCOVERY AND DATA MINING, KDD 2023, 2023, : 2694 - 2704
  • [27] Exploiting Simulated User Feedback for Conversational Search: Ranking, Rewriting, and Beyond
    Owoicho, Paul
    Sekulic, Ivan
    Aliannejadi, Mohammad
    Dalton, Jeffrey
    Crestani, Fabio
    PROCEEDINGS OF THE 46TH INTERNATIONAL ACM SIGIR CONFERENCE ON RESEARCH AND DEVELOPMENT IN INFORMATION RETRIEVAL, SIGIR 2023, 2023, : 632 - 642
  • [28] Designing and evaluating virtual musical instruments: facilitating conversational user interaction
    Johnston, Andrew
    Candy, Linda
    Edmonds, Ernest
    DESIGN STUDIES, 2008, 29 (06) : 556 - 571
  • [29] Enhancing Reflective and Conversational User Engagement in Argumentative Dialogues with Virtual Agents
    Aicher, Annalena
    Matsuda, Yuki
    Yasumoto, Keichii
    Minker, Wolfgang
    Andre, Elisabeth
    Ultes, Stefan
    MULTIMODAL TECHNOLOGIES AND INTERACTION, 2024, 8 (08)
  • [30] Designing Conversational Agents for Student An Exploratory Study of User Acceptance and Expectations
    Wang, Jieyu
    Zhang, Li
    Kang, Dingfang
    Pattit, Katherina G.
    INTERNATIONAL JOURNAL OF ADVANCED COMPUTER SCIENCE AND APPLICATIONS, 2024, 15 (10) : 43 - 52
12345