How many participants are really enough for usability studies?

Roobaea Alroobaea, Pam J. Mayhew

Research output: Chapter in Book/Report/Conference proceedingConference contribution

33 Citations (Scopus)

Abstract

The growth of the Internet and related technologies has enabled the development of a new breed of dynamic websites, applications and software products that are growing rapidly in use and that have had a great impact on many businesses. These technologies need to be continuously evaluated by usability evaluation methods (UEMs) to measure their efficiency and effectiveness, to assess user satisfaction, and ultimately to improve their quality. However, estimating the sample sizes for these methods has become the source of considerable debate at usability conferences. This paper aims to determine an appropriate sample size through empirical studies on the social network and educational domains by employing three types of UEM; it also examines further the impact of sample size on the findings of usability tests. Moreover, this paper quantifies the sample size required for the Domain Specific-to-context Inspection (DSI) method, which itself is developed through an adaptive framework. The results show that there is no certain number of participants for finding all usability problems; however, the rule of 16 4 users gains much validity in user testing. The magic number of five evaluators fails to find 80% of problems in heuristic evaluation, whereas three evaluators are enough to find 91% of usability problems in the DSI method.

Original languageEnglish
Title of host publicationProceedings of 2014 Science and Information Conference, SAI 2014
Place of PublicationNEW YORK
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages48-56
Number of pages9
ISBN (Print)9780989319317
DOIs
Publication statusPublished - 7 Oct 2014
Event2014 Science and Information Conference, SAI 2014 - London, United Kingdom
Duration: 27 Aug 201429 Aug 2014

Conference

Conference2014 Science and Information Conference, SAI 2014
CountryUnited Kingdom
CityLondon
Period27/08/1429/08/14

Keywords

  • Domain Specific Inspection (DSI)
  • Heuristic evaluation (HE)
  • methodological framework
  • sample size
  • User Testing (UT)

Cite this