James R. Lewis
Skip Abstract Section
Abstract
"Really, how many users do you need to test? Three answers, all different."---User Experience, Vol. 4, Issue 4, 2005
- Al-Awar, J., Chapanis, A., & Ford, R. (1981). Tutorials for the first-time computer user. IEEE Transactions on Professional Communication, 24, 30-37.Google Scholar
Cross Ref
- ANSI. (2001). Common industry format for usability test reports (ANSI-NCITS 354-2001). Washington, DC: American National Standards Institute.Google Scholar
- Bradley, J. V. (1968). Distribution-free statistical tests. Englewood Cliffs, NJ: Prentice-Hall.Google Scholar
- Bradley, J. V. (1976). Probability; decision; statistics. Englewood Cliffs, NJ: Prentice-Hall.Google Scholar
- Chapanis, A. (1981). Evaluating ease of use. Unpublished manuscript prepared for IBM, available on request from J. R. Lewis.Google Scholar
- Diamond, W. J. (1981). Practical experiment designs for engineers and scientists. Belmont, CA: Lifetime Learning Publications.Google Scholar
- Hertzum, M., & Jacobsen, N. J. (2003). The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction, 15, 183-204.Google ScholarCross Ref
- ISO. (1998). Ergonomic requirements for office work with visual display terminals (VDTs) - Part 11: Guidance on usability (ISO 9241-11:1998(E)). Geneva, Switzerland: Author.Google Scholar
- Kraemer, H. C., & Thiemann, S. (1987). How many subjects? Statistical power analysis in research. Newbury Park, CA: Sage.Google Scholar
- Lewis, J. R. (1982). Testing small system customer set-up. In Proceedings of the Human Factors Society 26th Annual Meeting (pp. 718-720). Santa Monica, CA: Human Factors Society.Google ScholarCross Ref
- Lewis, J. R. (1993). Problem discovery in usability studies: A model based on the binomial probability formula. In Proceedings of the Fifth International Conference on Human-Computer Interaction (pp. 666-671). Orlando, FL: Elsevier.Google Scholar
- Lewis, J. R. (1994). Sample sizes for usability studies: Additional considerations. Human Factors, 36, 368-378.Google ScholarCross Ref
- Lewis, J. R. (2001). Evaluation of procedures for adjusting problem-discovery rates estimated from small samples. International Journal of Human-Computer Interaction, 13, 445-479Google ScholarCross Ref
- Lewis, J. R. (2006). Usability testing. In G. Salvendy (ed.), Handbook of Human Factors and Ergonomics (pp. 1275-1316). New York, NY: John Wiley.Google Scholar
- Nielsen, J., & Landauer, T.K. (1993). A mathematical model of the finding of usability problems. In Proceedings of ACM INTERCHI'93 Conference (pp. 206-213). Amsterdam, Netherlands: ACM Press. Google ScholarDigital Library
- Nielsen, J., & Molich, R. (1990). Heuristic evaluation of user interfaces. In Conference Proceedings on Human Factors in Computing Systems - CHI90 (pp. 249-256). New York, NY: ACM. Google ScholarDigital Library
- Perfetti, C., & Landesman, L. (2001). Eight is not enough. Retrieved July 4, 2006 from http://www.uie.com/articles/eight_is_not_enough/Google Scholar
- Sauro, J. (2006). UI problem discovery sample size. Downloaded from Measuring Usability website, July 20, 2006-http://www.measuringusability.com/samplesize/problem_discovery.php.Google Scholar
- Spool, J., & Schroeder, W. (2001). Testing web sites: Five users is nowhere near enough. In CHI 2001 Extended Abstracts (pp. 285- 286). New York: ACM Press. Google ScholarDigital Library
- Turner, C. W., Lewis, J. R., & Nielsen, J. (2006). Determining usability test sample size. In W. Karwowski (ed.), International Encyclopedia of Ergonomics and Human Factors (pp. 3084-3088). Boca Raton, FL: CRC Press.Google Scholar
- Virzi, R. A. (1990). Streamlining the design process: Running fewer subjects. In Proceedings of the Human Factors Society 34th Annual Meeting (pp. 291-294). Santa Monica, CA: Human Factors Society.Google ScholarCross Ref
- Virzi, R.A. (1992). Refining the test phase of usability evaluation: How many subjects is enough? Human Factors, 34, 457-468. Google ScholarDigital Library
- Walpole, R. E. (1976). Elementary statistical concepts. New York, NY: Macmillan.Google Scholar
- Wixon, D. (2003). Evaluating usability methods: Why the current literature fails the practitioner. interactions, 10(4), 28-34. Google ScholarDigital Library
- Wright, P. C., & Monk, A. F. (1991). A cost-effective evaluation method for use by designers. International Journal of Man-Machine Studies, 35, 891-912. Google ScholarDigital Library
Index Terms
- Sample sizes for usability tests: mostly math, not magic
Recommendations
Usability tests refine requirements (abstract)
CSC '90: Proceedings of the 1990 ACM annual conference on CooperationThe traditional approach to developing a computer system is to gather requirements, write the specifications, code the programs, write the documentation, test, distribute, then wait for the accolades. Sometimes the accolades are long in coming. ...
Discourse Variations Between Usability Tests and Usability Reports
While usability evaluation and usability testing has become an important tool in artifact assessment, little is known about what happens to usability data as it moves from usability session to usability report. In this ethnographic case study, I ...
Cultural usability tests - how usability tests are not the same all over the world
UI-HCII'07: Proceedings of the 2nd international conference on Usability and internationalizationThe cultural diversity of users of technology challenges our methods for usability evaluation. In this paper we report on a multi-site, cross-cultural grounded theory field study of think aloud testing in seven companies in three countries (Denmark, ...
Comments