Sanne Elling
ABSTRACT
The retrospective think-aloud method, in which participants work in silence and verbalize their thoughts afterwards while watching a recording of their performance, is often used for the evaluation of websites. However, participants may not always be able to recall what they thought, when they only see few visual cues that help them remembering their task execution process. In our study we complemented the recording of the performance with a gaze trail of the participant" eye movements, in order to elicit more verbalizations. A comparison was made between the traditional retrospective think-aloud protocols and the variant with eye movements. Contrary to our expectations, no differences were found between the two conditions on numbers of problems, the ways these problems were detected, and types of problems. Two possible explanations for this result are that eye movements might be rather confronting and distracting for participants, and the rather generic way of probing we used. The added value might be stronger when specific questions are asked, based on the observed eye movements. Implications for usability practitioners are discussed in the conclusions of this paper.
- Ball, L.J., Eger, N., Stevens, R., Dodd, J. Applying the PEEP method in usability testing. Interfaces, 67 (2006), 15--19.Google Scholar
- Boren, T., and Ramey, J. Thinking aloud: Reconciling theory and practice. IEEE Transactions on Professional Communication 43, 3 (2000), 261--278.Google ScholarCross Ref
- Cooke, L. Is eye tracking the next step in usability testing? International professional communication conference (2006), 236--242.Google Scholar
- Cooke, L. Eye tracking: How it works and how it relates to usability. Technical Communication 52, 4 (2005), 456--463.Google Scholar
- Cooke, L., and Cuddihy, E. Using eye tracking to address limitations in think-aloud protocol. IEEE international professional communication conference proceedings (2005), 653--658.Google ScholarCross Ref
- Eger, N., Ball, L.J., Stevens, R., and Dodd, J. Cueing retrospective verbal reports in usability testing through eye-movement replay. Proceedings of the 21st British CHI group annual conference on HCI 2007: People and computers XXI: HCI... but not as we know it, Volume 1, British Computer Society (2007), 129--137. Google ScholarDigital Library
- Ehmke, C., and Wilson, S. Identifying web usability problems from eye-tracking data. Proceedings of the 21st British CHI group annual conference on HCI 2007: People and computers XXI: HCI... but not as we know it, Volume 1, British Computer Society (2007), 119--128. Google ScholarDigital Library
- Ericsson, K.A., and Simon, H.A. Protocol analysis: Verbal reports as data (Revised ed.). MIT Press, Cambridge, MA, 1996.Google Scholar
- Goldberg, J.H., and Wichansky, A.M. Eye tracking in usability evaluation: A practitioner's guide. In: The mind's eye: Cognitive and applied aspects of eye movement research. Hyönä, J., Radach, R., and Deubel, H., eds. Elsevier (2003), 493--516.Google ScholarCross Ref
- Guan, Z., Lee, S., Cuddihy, E., and Ramey, J. The validity of the stimulated retrospective think-aloud method as measured by eye tracking. Proceedings of the SIGCHI conference on human factors in computing systems. Montréal, Québec, Canada. ACM (2006), 1253--1262. Google ScholarDigital Library
- Hansen, J.P. The use of eye mark recordings to support verbal retrospection in software testing. Acta Psychologica 76 (1991), 31--49.Google ScholarCross Ref
- Hertzum, M., Hansen, K.D., and Andersen, H.H.K. Scrutinising usability evaluation: Does thinking aloud affect behaviour and mental workload? Behaviour & Information Technology 28, 2 (2009), 165--181. Google ScholarDigital Library
- Hornbæk, K. Dogmas in the assessment of usability evaluation methods. Behaviour & Information Technology 29, 1 (2010), 97--111. Google ScholarDigital Library
- Hyrskykari, A., Ovaska, S., Räihä, K., Majaranta, P., and Lehtinen, M. Gaze path stimulation in retrospective think-aloud. Journal of Eye Movement Research 2, 4 (2008), 1--18.Google ScholarCross Ref
- Jacob, R.J.K., and Karn, K.S. Eye tracking in human-computer interaction and usability research: Ready to deliver the promises. In: The mind's eye: Cognitive and applied aspects of eye movement research. Hyönä, J., Radach, R., and Deubel, H., eds. Elsevier Science BV (2003), 574--605.Google ScholarCross Ref
- Just, M.A., and Carpenter, P.A. Eye fixations and cognitive processes. Cognitive Psychology 8, 4 (1976), 441--480.Google ScholarCross Ref
- Krahmer, E., and Ummelen, N. Thinking about thinking aloud: A comparison of two verbal protocols for usability testing. IEEE Transactions on Professional Communication 47, 2, (2004), 105--117.Google ScholarCross Ref
- Lewis, C., and Mack, R. Learning to use a text processing system: Evidence from "thinking aloud" protocols. Proceedings of the 1982 conference on Human factors in computing systems. New York, USA. ACM (1982), 387--392. Google ScholarDigital Library
- Molich, R., and Dumas, J. Comparative usability evaluation (CUE-4). Behaviour & Information Technology 27, 3 (2008), 263--281. Google ScholarDigital Library
- Nørgaard, M., and Hornbæk, K. What do usability evaluators do in practice? An explorative study of think-aloud testing. Proceedings of the 6th conference on designing interactive systems ACM. (2006), 209--218. Google ScholarDigital Library
- Olmsted-Hawala, E.L., Murphy, E.D., Hawala, S., and Ashenfelter, K.T. Think-aloud protocols: A comparison of three think-aloud protocols for use in testing data-dissemination web sites for usability. Proceedings of the 28th international conference on human factors in computing systems ACM. (2010), 2381--2390. Google ScholarDigital Library
- Poole, A., and Ball, L.J. Eye tracking in human-computer interaction and usability research: Current status and future prospects. Encyclopedia of Human Computer Interaction (2005), 211--219.Google Scholar
- Ramey, J., Boren, T., Cuddihy, E., Dumas, J., Guan, Z., Van den Haak, M. J., and De Jong, M. D. T. Does think aloud work? How do we know? CHI'06 extended abstracts on human factors in computing systems ACM. (2006). Google ScholarDigital Library
- Renshaw, J.A., Finlay J., and Webb, N. Getting a measure of satisfaction from eyetracking in practice. CHI'06 extended abstracts on human factors in computing systems ACM. (2006). Google ScholarDigital Library
- Russo, J.E. A software system for the collection of retrospective protocols prompted by eye fixations. Behavior Research Methods & Instrumentation 11 (1979), 177--179.Google Scholar
- Ummelen, N., and Neutelings, R. Measuring reading behavior in policy documents: A comparison of two instruments. IEEE Transactions on Professional Communication 43, 3 (2000), 292--301.Google ScholarCross Ref
- Van den Haak, M.J., De Jong, M.D.T., and Schellens, P.J. Retrospective vs. concurrent think-aloud protocols: testing the usability of an online library catalogue. Behaviour & Information Technology 22, 5 (2003), 339--351.Google Scholar
- Van den Haak, M.J., De Jong, M.D.T., and Schellens, P.J. Employing think-aloud protocols and constructive interaction to test the usability of online library catalogues: A methodological comparison. Interacting with Computers 16, 6 (2004), 1153--1170.Google ScholarCross Ref
- Van den Haak, M.J., De Jong, M.D.T., and Schellens, P.J. Evaluation of an informational web site: Three variants of the think-aloud method compared. Technical Communication 54, 1 (2007), 58--71.Google Scholar
- Van den Haak, M.J., De Jong, M.D.T., and Schellens, P.J. Evaluating municipal websites: A methodological comparison of three think-aloud variants. Government Information Quarterly 26, 1 (2009), 193--202..Google ScholarCross Ref
Index Terms
- Retrospective think-aloud method: using eye movements as an extra cue for participants' verbalizations
Recommendations
Concurrent Think-Aloud Verbalizations and Usability Problems
The concurrent think-aloud protocol—in which participants verbalize their thoughts when performing tasks—is a widely employed approach in usability testing. Despite its value, analyzing think-aloud sessions can be onerous because it often entails ...
The validity of the stimulated retrospective think-aloud method as measured by eye tracking
CHI '06: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsRetrospective Think aloud (RTA) is a usability method that collects the verbalization of a user's performance after the performance is over. There has been little work done to investigate the validity and reliability of RTA. This paper reports on an ...
Think-aloud protocols: a comparison of three think-aloud protocols for use in testing data-dissemination web sites for usability
CHI '10: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsWe describe an empirical, between-subjects study on the use of think-aloud protocols in usability testing of a federal data-dissemination Web site. This double-blind study used three different types of think-aloud protocols: a traditional protocol, a ...
Comments