On the Reliability of User Feedback for Evaluating the Quality of Conversational Agents

被引：0
|
作者：
Massiah, Jordan [1 ,2 ]
Yilmaz, Emine [1 ,2 ]
Jiao, Yunlong [1 ]
Kazai, Gabriella [1 ]
机构：
[1] Amazon, London, England
[2] UCL, London, England
来源：
PROCEEDINGS OF THE 32ND ACM INTERNATIONAL CONFERENCE ON INFORMATION AND KNOWLEDGE MANAGEMENT, CIKM 2023 | 2023年
关键词：
conversational agent quality; user feedback reliability;
D O I：
10.1145/3583780.3615286
中图分类号：
TP18 [人工智能理论];
学科分类号：
081104 ; 0812 ; 0835 ; 1405 ;
摘要：
We analyse the reliability of users' explicit feedback for evaluating the quality of conversational agents. Using data from a commercial conversational system, we analyse how user feedback compares with human annotations; how well it aligns with implicit user satisfaction signals, such as retention; and how much user feedback is needed to reliably evaluate the quality of a conversational system.
引用
收藏
页码：4185 / 4189
页数：5
相关论文
共 50 条
  • [41] User's Utterance Classification Using Machine Learning for Arabic Conversational Agents
    Hijjawi, Mohammad
    Bandar, Zuhair
    Crockett, Keeley
    2013 5TH INTERNATIONAL CONFERENCE ON COMPUTER SCIENCE AND INFORMATION TECHNOLOGY (CSIT), 2013, : 223 - 232
  • [42] Designing a Study for Evaluating User Feedback on Predesign Models
    Voehringer, Juergen
    Bellstroem, Peter
    Gaelle, Doris
    Kop, Christian
    INFORMATION SYSTEMS DEVELOPMENT: ASIAN EXPERIENCES, 2011, : 411 - 423
  • [43] A Neural Network Approach to Intention Modeling for User-Adapted Conversational Agents
    Griol, David
    Callejas, Zoraida
    COMPUTATIONAL INTELLIGENCE AND NEUROSCIENCE, 2016, 2016
  • [44] Intelligent Conversational Agents in Mental Healthcare Services: A Thematic Analysis of User Perceptions
    Prakash, Ashish Viswanath
    Das, Saini
    PACIFIC ASIA JOURNAL OF THE ASSOCIATION FOR INFORMATION SYSTEMS, 2020, 12 (02): : 1 - 34
  • [45] 'user as assessor' approach to embodied conversational agents - The case of apparent attention in ECAs
    Nass, C
    Robles, E
    Wang, QY
    FROM BROWS TO TRUST: EVALUATING EMBODIED CONVERSATIONAL AGENTS, 2004, 7 : 161 - 188
  • [46] A User Perception-Based Approach to Create Smiling Embodied Conversational Agents
    Ochs, Magalie
    Pelachaud, Catherine
    Mckeown, Gary
    ACM TRANSACTIONS ON INTERACTIVE INTELLIGENT SYSTEMS, 2017, 7 (01)
  • [47] Embodied Conversational Agents as Conversational Partners
    Louwerse, Max M.
    Graesser, Arthur C.
    McNamara, Danielle S.
    Lu, Shulan
    APPLIED COGNITIVE PSYCHOLOGY, 2009, 23 (09) : 1244 - 1255
  • [48] Quality index for feedback and reliability improvement
    Jain, AK
    ASQ'S 57TH ANNUAL QUALITY CONGRESS PROCEEDINGS: EXPANDING HORIZON: GLOBAL, PERSONAL, TOOLS, NETWORKING, SOLUTIONS, 2003, : 305 - 311
  • [49] Conversational user interfaces
    Wahlster, Wolfgang
    IT - Information Technology, 2004, 46 (06): : 289 - 290
  • [50] Exquisitor at the Lifelog Search Challenge 2024: Blending Conversational Search with User Relevance Feedback
    Khan, Omar Shahbaz
    Sharma, Ujjwal
    Zhu, Hongyi
    Rudinac, Stevan
    Jonsson, Bjorn Por
    PROCEEDINGS OF 2024 ACM WORKSHOP ON THE LIFELOG SEARCH CHALLENGE, LSC 2024, 2024, : 117 - 121
12345