On the Reliability of User Feedback for Evaluating the Quality of Conversational Agents

被引：0
|
作者：
Massiah, Jordan [1 ,2 ]
Yilmaz, Emine [1 ,2 ]
Jiao, Yunlong [1 ]
Kazai, Gabriella [1 ]
机构：
[1] Amazon, London, England
[2] UCL, London, England
来源：
PROCEEDINGS OF THE 32ND ACM INTERNATIONAL CONFERENCE ON INFORMATION AND KNOWLEDGE MANAGEMENT, CIKM 2023 | 2023年
关键词：
conversational agent quality; user feedback reliability;
D O I：
10.1145/3583780.3615286
中图分类号：
TP18 [人工智能理论];
学科分类号：
081104 ; 0812 ; 0835 ; 1405 ;
摘要：
We analyse the reliability of users' explicit feedback for evaluating the quality of conversational agents. Using data from a commercial conversational system, we analyse how user feedback compares with human annotations; how well it aligns with implicit user satisfaction signals, such as retention; and how much user feedback is needed to reliably evaluate the quality of a conversational system.
引用
收藏
页码：4185 / 4189
页数：5
相关论文
共 50 条
  • [31] Leveraging User Simulation to Develop and Evaluate Conversational Information Access Agents
    Bernard, Nolwenn
    PROCEEDINGS OF THE 17TH ACM INTERNATIONAL CONFERENCE ON WEB SEARCH AND DATA MINING, WSDM 2024, 2024, : 1136 - 1138
  • [32] Conversational Agents Trust Calibration A User-Centred Perspective to Design
    Dubiel, Mateusz
    Daronnat, Sylvain
    Leiva, Luis A.
    PROCEEDINGS OF THE 4TH INTERNATIONAL CONFERENCE ON CONVERSATIONAL USER INTERFACES, CUI 2022, 2022,
  • [33] Agent Simulation to Develop Interactive and User-Centered Conversational Agents
    Griol, David
    Carbo, Javier
    Molina, Jose M.
    INTERNATIONAL SYMPOSIUM ON DISTRIBUTED COMPUTING AND ARTIFICIAL INTELLIGENCE, 2011, 91 : 69 - 76
  • [34] Conversational agents and momentary user experience: an assessment using an electroencephalography device
    Brock, Lais Andressa
    De Bortoli, Lis Angela
    Bellei, Ericles Andrei
    De Marchi, Ana Carolina Bertoletti
    UNIVERSAL ACCESS IN THE INFORMATION SOCIETY, 2024,
  • [35] Critical information quality dimensions of conversational agents for healthcare
    Liu, Caihua
    Peng, Guochao
    Kong, Shufeng
    Lan, Chaowang
    Zhu, Haoliang
    INFORMATION RESEARCH-AN INTERNATIONAL ELECTRONIC JOURNAL, 2023, 28 (04): : 18 - 42
  • [36] Incorporating User Feedback in Conversational Question Answering over Heterogeneous Web Sources
    Kaiser, Magdalena
    PROCEEDINGS OF THE 43RD INTERNATIONAL ACM SIGIR CONFERENCE ON RESEARCH AND DEVELOPMENT IN INFORMATION RETRIEVAL (SIGIR '20), 2020, : 2482 - 2482
  • [37] Adapting User Preference to Online Feedback in Multi-round Conversational Recommendation
    Xu, Kerui
    Yang, Jingxuan
    Xu, Jun
    Gao, Sheng
    Guo, Jun
    Wen, Ji-Rong
    WSDM '21: PROCEEDINGS OF THE 14TH ACM INTERNATIONAL CONFERENCE ON WEB SEARCH AND DATA MINING, 2021, : 364 - 372
  • [38] user2agent: 2nd Workshop on User-Aware Conversational Agents
    Shmueli-Scheuer, Michal
    Artstein, Ron
    Khazaeni, Yasaman
    Fang, Hao
    Liao, Q. Vera
    PROCEEDINGS OF THE 25TH INTERNATIONAL CONFERENCE ON INTELLIGENT USER INTERFACES COMPANION (IUI'20), 2020, : 9 - 10
  • [39] Evaluating Mixed-initiative Conversational Search Systems via User Simulation
    Sekulic, Ivan
    Aliannejadi, Mohammad
    Crestani, Fabio
    WSDM'22: PROCEEDINGS OF THE FIFTEENTH ACM INTERNATIONAL CONFERENCE ON WEB SEARCH AND DATA MINING, 2022, : 888 - 896
  • [40] Evaluating Unsupervised Text Embeddings on Software User Feedback
    Devine, Peter
    Koh, Yun Sing
    Blincoe, Kelly
    29TH IEEE INTERNATIONAL REQUIREMENTS ENGINEERING CONFERENCE WORKSHOPS (REW 2021), 2021, : 87 - 95
12345