To intervene or not to intervene: An investigation of three think-aloud protocols in usability testing

This paper presents the results of a study investigating the use of three think-aloud methods in website usability testing: the concurrent think-aloud, the speech-communication, and the active intervention methods. These three methods were compared through an evaluation of a library website, which involved four points of comparison: overall task performance, test participants’ experiences, the quantity and quality of usability problems discovered, and the cost of employing the methods. Data were collected from 60 individuals, with 20 participants allocated to each testing method, who were asked to complete a set of nine experimental tasks. The results of the study revealed that the three variations enabled the identification of a similar number of usability problems and types. However, the active intervention method was found to cause some reactivity, modifying participants’ interaction with the interface, and negatively affecting their feelings towards the evaluator. The active intervention method also required much greater investment than did the other two methods in terms of evaluators' time.