Consolidation of Usability Problems With Novice Evaluators Re-Examined in Individual vs. Collaborative Settings

Usability testing can involve multiple users and evaluators. In such cases, consolidating usability problems (UPs) constitutes an essential part of data analysis. In a between-subjects design, this study aims to re-examine a previous study by comparing the results of novice evaluators merging UPs individually vs. collaboratively and to assess the quality of the final UP lists, by computing the merging rate and the accuracy rate, respectively. Law and Hvannberg compared the results of evaluators merging UPs individually vs. collaboratively in a within-subjects design, revealing a tendency towards merging UPs in collaborative settings. In the present study, 45 novice evaluators consolidated four UP lists into a single UP master list while working alone or with a partner. The results showed no significant difference between evaluators in the two settings, suggesting that the UP consolidation process does not benefit from positive group decision effects. RESEARCH HIGHLIGHTS Based on a between-subjects design, the study results indicate no difference in merging rate or quality of consolidation output between novice evaluators consolidating individually or collaboratively. Working together neither helps nor impairs novices during the consolidation activity. The study showed that collaborative UP consolidation takes more time than individual consolidation without resulting in higher quality. Thus, investing in more manpower of novices does not appear to be beneficial. According to group performance models, the lack of performance gains in the collaborative setting, as found in the current study, could stem partly from the use of participants with little evaluation experience.