Abstract
A lognormal model for response times is used to check response times for aberrances in examinee behavior on computerized adaptive tests. Both classical procedures and Bayesian posterior predictive checks are presented. For a fixed examinee, responses and response times are independent; checks based on response times offer thus information independent of the results of checks on response patterns. Empirical examples of the use of classical and Bayesian checks for detecting two different types of aberrances in response times are presented. The detection rates for the Bayesian checks outperformed those for the classical checks, but at the cost of higher false-alarm rates. A guideline for the choice between the two types of checks is offered.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bradlow, E.T., Weiss, R. E., & Cho, M. (1998). Bayesian detection of outliers in computerized adaptive tests.Journal of the American Statistical Association, 93, 910–919.
Drasgow, F., Levine, M.V., Williams, E.A. (1985). Appropriateness measurement with polytomous item reponse models and standardized indices.British Journal of Mathematical and Statistical Psychology, 38, 67–86.
Gelman, A., Carlin, J.B., Stern, H., & Rubin, D.B. (1995).Bayesian data analysis. London, U.K.: Chapman & Hall.
Johnson, V.E., & Albert, J.H. (1999).Ordinal data modeling. New York, NY: Springer-Verlag.
Levine, M.V., & Rubin, D.B. (1979). Measuring the appropriateness of multiple-choice test scores.Journal of Educational Statistics, 4, 269–290.
Meijer, R.R., & Sijtsma, K. (1995). Detection of aberrant item response patterns: A review of recent developments.Applied Measurement in Education, 8, 261–272.
Mislevy, R.J., & Chang, H. (2000). Does adaptive testing violate local independence?Psychometrika, 65, 149–156.
Mislevy, R.J., & Wu, P.-K. (1996).Missing responses and Bayesian IRT estimation: Omits, choice, time limits, and adaptive testing (Research Rep. RR-96-30-ONR). Princeton, NJ: Educational Testing Service.
Molenaar, I.W., & Hoijtink, H. (1990). The many null distributions of person-fit statistics.Psychometrika, 55, 75–106.
Neter, J., Wasserman, W., & Kutner, M.H. (1985).Applied linear statistical models: Regression, analysis of variance, and experimental designs. Homewood, IL: Richard D. Irwin.
Segall, D.O., Moreno, K.E., & Hetter, D.H. (1997). In W.A. Sands, B.K. Waters, & J.R. McBride (Eds.),Computerized adaptive testing: From inquiry to operation (pp. 117–130). Washington, DC: American Psychological Association.
Schnipke, D.L., & Scrams, D.J. (1997).Representing response time information in item banks (LSAC Computerized Testing Rep. No. 97-09). Newtown, PA: Law School Admission Council.
Thissen, D. (1983). Timed testing: An approach using item resonse theory. In D.J. Weiss (Ed.), New horizons in testing: Latent trait test theory and computerized adaptive testing (pp. 179–203). New York, NY: Academic Press.
Trabin, T.E., & Weiss, D.J. (1983). The person response curve: Fit of individuals to item response theory models. In D.J. Weiss (Ed.),New horizons in testing: Latent trait theory and computerized adaptive testing. New York, NY: Academic Press.
van der Linden, W.J. (2002).A model for speed and accuracy on tests. Unpublished manuscipt.
van der Linden, W.J., & Pashley, P.J. (2000). Item selection and ability estimation in adaptive testing. In W.J. van der Linden & C.A.W. Glas (Eds.),Computerized adaptive testing: Theory and practice (pp. 1–25). Norwell, MA: Kluwer Academic Publishers.
van der Linden, W.J., Scrams, D.J., & Schnipke, D.L. (1999). Using response-time constraints to control for speededness in computerized adaptive testing.Applied Psychological Measurement, 23, 195–210.
van Krimpen-Stoop, E.M.L.A., & Meijer, R.R. (1999). Simulating the null distribution of person-fit statistics for conventional and adaptive tests.Applied Psychological Measurement, 23, 327–345.
van Krimpen-Stoop, E.M.L.A., & Meijer, R.R. (2000). Detecting person misfit in adaptive testing using statistical process control techniques. In W.J. van der Linden & C.A.W. Glas. (Eds.),Computerized adaptive testing: Theory and practice (pp. 221–219). Norwell, MA: Kluwer Academic Publishers.
Author information
Authors and Affiliations
Corresponding author
Additional information
This study received funding from the Law School Admission Council (LSAC). The opinions and conclusions contained in this paper are those of the authors and do not necessarily reflect the policy and position of LSAC. The authors are most indebted to Wim M. M. Tielen for his computational assistance and to the US Defense Manpower Data Center for the permission to use the ASVAB data set in the empirical examples.
Rights and permissions
About this article
Cite this article
van der Linden, W.J., van Krimpen-Stoop, E.M.L.A. Using response times to detect aberrant responses in computerized adaptive testing. Psychometrika 68, 251–265 (2003). https://doi.org/10.1007/BF02294800
Received:
Revised:
Issue Date:
DOI: https://doi.org/10.1007/BF02294800