On the Reliability of User Feedback for Evaluating the Quality of Conversational Agents

被引：0
|
作者：
Massiah, Jordan [1 ,2 ]
Yilmaz, Emine [1 ,2 ]
Jiao, Yunlong [1 ]
Kazai, Gabriella [1 ]
机构：
[1] Amazon, London, England
[2] UCL, London, England
来源：
PROCEEDINGS OF THE 32ND ACM INTERNATIONAL CONFERENCE ON INFORMATION AND KNOWLEDGE MANAGEMENT, CIKM 2023 | 2023年
关键词：
conversational agent quality; user feedback reliability;
D O I：
10.1145/3583780.3615286
中图分类号：
TP18 [人工智能理论];
学科分类号：
081104 ; 0812 ; 0835 ; 1405 ;
摘要：
We analyse the reliability of users' explicit feedback for evaluating the quality of conversational agents. Using data from a commercial conversational system, we analyse how user feedback compares with human annotations; how well it aligns with implicit user satisfaction signals, such as retention; and how much user feedback is needed to reliably evaluate the quality of a conversational system.
引用
收藏
页码：4185 / 4189
页数：5
相关论文
共 50 条
  • [1] User-Aware Conversational Agents
    Liao, Q. Vera
    Shmueli-Scheuer, Michal
    Wen, Tsung-Hsien
    Yu, Zhou
    PROCEEDINGS OF THE 24TH INTERNATIONAL CONFERENCE ON INTELLIGENT USER INTERFACES: COMPANION (IUI 2019), 2019, : 133 - 134
  • [2] Exploring Effects of Conversational Fillers on User Perception of Conversational Agents
    Jeong, Yuin
    Lee, Juho
    Kang, Younah
    CHI EA '19 EXTENDED ABSTRACTS: EXTENDED ABSTRACTS OF THE 2019 CHI CONFERENCE ON HUMAN FACTORS IN COMPUTING SYSTEMS, 2019,
  • [3] Incremental multimodal feedback for conversational agents
    Kopp, Stefan
    Stocksmeier, Thorsten
    Gibbon, Dafydd
    INTELLIGENT VIRTUAL AGENTS, PROCEEDINGS, 2007, 4722 : 139 - +
  • [4] User Evaluation of Conversational Agents for Aerospace Domain
    Liu, Ying-Hsang
    Arnold, Alexandre
    Dupont, Gerard
    Kobus, Catherine
    Lancelot, Francois
    Granger, Geraud
    Rouillard, Yves
    Duchevet, Alexandre
    Imbert, Jean-Paul
    Matton, Nadine
    INTERNATIONAL JOURNAL OF HUMAN-COMPUTER INTERACTION, 2024, 40 (19) : 5549 - 5568
  • [5] USER-AWARENESS AND ADAPTATION IN CONVERSATIONAL AGENTS
    Delic, Vlado
    Gnjatovic, Milan
    Jakovljevic, Niksa
    Popovic, Branislav
    Jokic, Ivan
    Bojanic, Milana
    FACTA UNIVERSITATIS-SERIES ELECTRONICS AND ENERGETICS, 2014, 27 (03) : 375 - 387
  • [6] Research on the method of evaluating automobile reliability by the data from user's feedback
    Yan, Yong
    Cao, Zhengqing
    Nongye Jixie Xuebao/Transactions of the Chinese Society of Agricultural Machinery, 30 (03): : 93 - 96
  • [7] Modeling User's Neutral Feedback in Conversational Recommendation
    Li, Xizhe
    Hu, Chenhao
    Kong, Weiyang
    Zhang, Sen
    Liu, Yubao
    NEURAL INFORMATION PROCESSING, ICONIP 2023, PT IV, 2024, 14450 : 56 - 68
  • [8] Evaluating Conversational Recommender Systems via User Simulation
    Zhang, Shuo
    Balog, Krisztian
    KDD '20: PROCEEDINGS OF THE 26TH ACM SIGKDD INTERNATIONAL CONFERENCE ON KNOWLEDGE DISCOVERY & DATA MINING, 2020, : 1512 - 1520
  • [9] Evaluating Use Cases Suitability for Conversational User Interfaces
    Ferreira, Pedro
    Vasconcelos, Andre
    PROCEEDINGS OF THE 21ST INTERNATIONAL CONFERENCE ON ENTERPRISE INFORMATION SYSTEMS (ICEIS), VOL 1, 2019, : 431 - 437
  • [10] Information quality of conversational agents in healthcare
    Liu, Caihua
    Zowghi, Didar
    Peng, Guochao
    Kong, Shufeng
    INFORMATION DEVELOPMENT, 2023,
12345