Skip to main content
Top
Published in: Perspectives on Medical Education 6/2018

Open Access 01-12-2018 | Original Article

Students as stakeholders in assessment: how students perceive the value of an assessment

Authors: Michelle Ricci, Christina St-Onge, Jing Xiao, Meredith Young

Published in: Perspectives on Medical Education | Issue 6/2018

Login to get access

Abstract

Introduction

For assessment to fill an educational role, students must see the results generated by assessment as valuable, and actively engage with this feedback in order to support learning. Few studies include examinees as stakeholders in validation beyond general notions of acceptability. Here, we explore students as stakeholders in the validation of a newly implemented assessment.

Methods

A student-relevant validity framework based on the unified theory of validity was created and adapted to a survey format. Likert-style items were used to examine first- and second-year medical students’ perceptions of a new cumulative assessment, with several open-ended items. Analysis included: mean ratings per subscale of validity evidence, thematic analysis of comments, and a correlation between questionnaire subscores and exam performance.

Results

Seventy-seven students participated (20.5%). Student perceptions of the assessment were favourable, with significantly different ratings across validity evidence (Response Process (4.8 (SD = 0.7); scored/6), Content (4.6(0.9)), Consequential (4.4(0.8)), Internal Structure (4.2(0.9)), and Relationship to Other Variables (4.0(1.0))). Exam performance correlated with subscores for Relationship to Other Variables (r = 0.34, p < 0.005) and Response Process (r = 0.24, p < 0.05).

Discussion

Students perceived the assessment as facilitating learning, providing ‘checkpoints’, and were disappointed when it did not meet their expectations regarding the purpose of assessment. If students perceive that results do not reflect their future performance in clinical environments, or do not align with their perceived purpose of assessment, the educational value of assessment may be limited. It is critical to understand when, and how students engage in interpreting and integrating assessment-generated feedback to ensure that assessment contributes positively to learning.
Literature
1.
go back to reference Eva KW, Armson H, Holmboe E, et al. Factors influencing responsiveness to feedback: on the interplay between fear, confidence, and reasoning processes. Adv Health Sci Educ. 2012;17:15–26.CrossRef Eva KW, Armson H, Holmboe E, et al. Factors influencing responsiveness to feedback: on the interplay between fear, confidence, and reasoning processes. Adv Health Sci Educ. 2012;17:15–26.CrossRef
2.
go back to reference Nicol DJ, Macfarlane-Dick D. Formative assessment and self-regulated learning : a model and seven principles of good feedback practice. Stud High Educ. 2006;31:199–218.CrossRef Nicol DJ, Macfarlane-Dick D. Formative assessment and self-regulated learning : a model and seven principles of good feedback practice. Stud High Educ. 2006;31:199–218.CrossRef
3.
go back to reference Tamblyn R, Abrahamowicz M, Brailovsky C, et al. Association between licensing examination scores and resource use and quality of care in primary care practice. JAMA. 1998;280:989–96.CrossRef Tamblyn R, Abrahamowicz M, Brailovsky C, et al. Association between licensing examination scores and resource use and quality of care in primary care practice. JAMA. 1998;280:989–96.CrossRef
4.
go back to reference Tamblyn R, Abrahamowicz M, Dauphinee D, et al. Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities. JAMA. 2007;298:993–1001.CrossRef Tamblyn R, Abrahamowicz M, Dauphinee D, et al. Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities. JAMA. 2007;298:993–1001.CrossRef
5.
go back to reference Tamblyn R, Abrahamowicz M, Dauphinee WD, et al. Association between lincensure examination scores and practice in primary care. JAMA. 2002;288:3019–26.CrossRef Tamblyn R, Abrahamowicz M, Dauphinee WD, et al. Association between lincensure examination scores and practice in primary care. JAMA. 2002;288:3019–26.CrossRef
6.
go back to reference Boud D. Sustainable assessment: rethinking assessment for the learning society. Stud Contin Educ. 2000;22:151–67.CrossRef Boud D. Sustainable assessment: rethinking assessment for the learning society. Stud Contin Educ. 2000;22:151–67.CrossRef
7.
go back to reference Gipps CV. Beyond testing: towards a theory of educational assessment. Philadelphia, PA: Psychology Press; 1994. Gipps CV. Beyond testing: towards a theory of educational assessment. Philadelphia, PA: Psychology Press; 1994.
8.
go back to reference Shepard AL. The role of assessment in a learning culture. Educ Res. 2000;29:4–14.CrossRef Shepard AL. The role of assessment in a learning culture. Educ Res. 2000;29:4–14.CrossRef
9.
go back to reference Eva KW, Bordage G, Campbell C, et al. Towards a program of assessment for health professionals: from training into practice. Adv Health Sci Educ. 2016;21:897–913.CrossRef Eva KW, Bordage G, Campbell C, et al. Towards a program of assessment for health professionals: from training into practice. Adv Health Sci Educ. 2016;21:897–913.CrossRef
10.
go back to reference Martinez ME, Lipson JI. Assessment for learning. Educ Leadersh. 1989;46:73–5. Martinez ME, Lipson JI. Assessment for learning. Educ Leadersh. 1989;46:73–5.
11.
go back to reference Rushton A. Formative assessment: a key to deep learning? Med Teach. 2005;27:509–13.CrossRef Rushton A. Formative assessment: a key to deep learning? Med Teach. 2005;27:509–13.CrossRef
12.
go back to reference Schuwirth LWT, Van der Vleuten CPM. Programmatic assessment: From assessment of learning to assessment for learning. Med Teach. 2011;33:478–85.CrossRef Schuwirth LWT, Van der Vleuten CPM. Programmatic assessment: From assessment of learning to assessment for learning. Med Teach. 2011;33:478–85.CrossRef
13.
go back to reference Duffield K, Spencer J. A survey of medical students’ views about the purposes and fairness of assessment. Med Educ. 2002;36:879–86.CrossRef Duffield K, Spencer J. A survey of medical students’ views about the purposes and fairness of assessment. Med Educ. 2002;36:879–86.CrossRef
14.
go back to reference Norcini J, Anderson B, Bollela V, et al. Criteria for good assessment: consensus statement and recommendations from the ottawa 2010 conference. Med Teach. 2011;33:206–14.CrossRef Norcini J, Anderson B, Bollela V, et al. Criteria for good assessment: consensus statement and recommendations from the ottawa 2010 conference. Med Teach. 2011;33:206–14.CrossRef
15.
go back to reference Norcini JJ, McKinley DW. Assessment methods in medical education. Teach Teach Educ. 2007;23:239–50.CrossRef Norcini JJ, McKinley DW. Assessment methods in medical education. Teach Teach Educ. 2007;23:239–50.CrossRef
16.
go back to reference Van der Vleuten CPM. The assessment of professional competence: Developments, research and practical implications. Adv Health Sci Educ. 1996;1:41–67.CrossRef Van der Vleuten CPM. The assessment of professional competence: Developments, research and practical implications. Adv Health Sci Educ. 1996;1:41–67.CrossRef
17.
go back to reference Hill D, Guinea A, McCarthy W. Formative assessment: a student perspective. Med Educ. 1994;28:394–9.CrossRef Hill D, Guinea A, McCarthy W. Formative assessment: a student perspective. Med Educ. 1994;28:394–9.CrossRef
18.
go back to reference Stefani LA. Assessment in partnership with learners. Assess Eval High Educ. 1998;23:339–50.CrossRef Stefani LA. Assessment in partnership with learners. Assess Eval High Educ. 1998;23:339–50.CrossRef
19.
go back to reference Yorke M. Formative assessment in higher education: moves towards theory and the enhancement of pedagogic practice. High Educ. 2003;45:477.CrossRef Yorke M. Formative assessment in higher education: moves towards theory and the enhancement of pedagogic practice. High Educ. 2003;45:477.CrossRef
20.
go back to reference Epstein RM. Assessment in medical education. N Engl J Med. 2007;356:387–96.CrossRef Epstein RM. Assessment in medical education. N Engl J Med. 2007;356:387–96.CrossRef
21.
go back to reference Barclay LJ, Skarlicki DP, Pugh SD. Exploring the role of emotions in injustice perceptions and retaliation. J Appl Psychol. 2005;90:629.CrossRef Barclay LJ, Skarlicki DP, Pugh SD. Exploring the role of emotions in injustice perceptions and retaliation. J Appl Psychol. 2005;90:629.CrossRef
22.
go back to reference De Cremer D, Van Knippenberg B, Van Knippenberg D, Mullenders D, Stinglhamber F. Rewarding leadership and fair procedures as determinants of self-esteem. J Appl Psychol. 2005;90:3.CrossRef De Cremer D, Van Knippenberg B, Van Knippenberg D, Mullenders D, Stinglhamber F. Rewarding leadership and fair procedures as determinants of self-esteem. J Appl Psychol. 2005;90:3.CrossRef
23.
go back to reference Higgins RS, Bridges J, Burke JM, O’Donnell MA, Cohen NM, Wilkes SB. Implementing the ACGME general competencies in a cardiothoracic surgery residency program using 360-degree feedback. Ann Thorac Surg. 2004;77:12–7.CrossRef Higgins RS, Bridges J, Burke JM, O’Donnell MA, Cohen NM, Wilkes SB. Implementing the ACGME general competencies in a cardiothoracic surgery residency program using 360-degree feedback. Ann Thorac Surg. 2004;77:12–7.CrossRef
24.
go back to reference Messick S. Validity. In: Linn RL, editor. Educational measurement. New York: Macmillan; 1989. pp. 13–103. Messick S. Validity. In: Linn RL, editor. Educational measurement. New York: Macmillan; 1989. pp. 13–103.
25.
go back to reference American Educational Research Association, American Psychological Association. National Council on Measurement in Education Joint Committee on Standards for Educational and Psychological Testing. Standards for educational and psychological testing. Washington, DC: AERA; 1999. American Educational Research Association, American Psychological Association. National Council on Measurement in Education Joint Committee on Standards for Educational and Psychological Testing. Standards for educational and psychological testing. Washington, DC: AERA; 1999.
26.
go back to reference Downing SM. Validity: on the meaningful interpretation of assessment data. Med Educ. 2003;37:830–7.CrossRef Downing SM. Validity: on the meaningful interpretation of assessment data. Med Educ. 2003;37:830–7.CrossRef
27.
go back to reference Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med. 2006;119:166–166e7.CrossRef Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med. 2006;119:166–166e7.CrossRef
28.
go back to reference Kane MT. Validating the interpretations and uses of test scores. J Educ Meas. 2013;50:1–73.CrossRef Kane MT. Validating the interpretations and uses of test scores. J Educ Meas. 2013;50:1–73.CrossRef
29.
go back to reference DeVellis RF. Scale development: theory and applications. Vol. 26. Thousand Oaks: SAGE; 2012. DeVellis RF. Scale development: theory and applications. Vol. 26. Thousand Oaks: SAGE; 2012.
30.
go back to reference Mundfrom DJ, Shaw DG, Ke LT. Minimum sample size recommendations for conducting factor analysis. Int J Testing. 2005;5:159–68.CrossRef Mundfrom DJ, Shaw DG, Ke LT. Minimum sample size recommendations for conducting factor analysis. Int J Testing. 2005;5:159–68.CrossRef
31.
go back to reference Sandelowski M. Focus on research methods-whatever happened to qualitative description? Res Nurs Health. 2000;23:334–40.CrossRef Sandelowski M. Focus on research methods-whatever happened to qualitative description? Res Nurs Health. 2000;23:334–40.CrossRef
32.
go back to reference Greene JC. Is mixed methods social inquiry a distinctive methodology? J Mix Methods Res. 2008;2:7–22.CrossRef Greene JC. Is mixed methods social inquiry a distinctive methodology? J Mix Methods Res. 2008;2:7–22.CrossRef
33.
go back to reference Ebel RL, Frisbie DA. Essentials of educational measurement. Vol. 5. Englewood Cliffs: Prentice-Hall; 1991. Ebel RL, Frisbie DA. Essentials of educational measurement. Vol. 5. Englewood Cliffs: Prentice-Hall; 1991.
34.
go back to reference Brett JF, Atwater LE. 360° feedback: accuracy, reactions, and perceptions of usefulness. J Appl Psychol. 2001;86:930.CrossRef Brett JF, Atwater LE. 360° feedback: accuracy, reactions, and perceptions of usefulness. J Appl Psychol. 2001;86:930.CrossRef
35.
go back to reference Dipboye RL, de Pontbriand R. Correlates of employee reactions to performance appraisals and appraisal systems. J Appl Psychol. 1981;66:248.CrossRef Dipboye RL, de Pontbriand R. Correlates of employee reactions to performance appraisals and appraisal systems. J Appl Psychol. 1981;66:248.CrossRef
36.
go back to reference Sargeant J, Mann K, Sinclair D, Van der Vleuten C, Metsemakers J. Understanding the influence of emotions and reflection upon multi-source feedback acceptance and use. Adv Health Sci Educ. 2008;13:275–88.CrossRef Sargeant J, Mann K, Sinclair D, Van der Vleuten C, Metsemakers J. Understanding the influence of emotions and reflection upon multi-source feedback acceptance and use. Adv Health Sci Educ. 2008;13:275–88.CrossRef
37.
go back to reference Gielen S, Dochy F, Dierick S. Evaluating the consequential validity of new modes of assessment: The influence of assessment on learning, including pre-, post-, and true assessment effects. In: Segers M, Dochy F, Cascallar E, editors. Optimising new modes of assessment: In search of qualities and standards. Dordrecht: Kluwer; 2003. pp. 37–54.CrossRef Gielen S, Dochy F, Dierick S. Evaluating the consequential validity of new modes of assessment: The influence of assessment on learning, including pre-, post-, and true assessment effects. In: Segers M, Dochy F, Cascallar E, editors. Optimising new modes of assessment: In search of qualities and standards. Dordrecht: Kluwer; 2003. pp. 37–54.CrossRef
38.
go back to reference Van der Vleuten CPM, Schuwirth LWT. Assessing professional competence: from methods to programmes. Med Educ. 2005;39:309–17.CrossRef Van der Vleuten CPM, Schuwirth LWT. Assessing professional competence: from methods to programmes. Med Educ. 2005;39:309–17.CrossRef
39.
go back to reference Watling CJ, Kenyon CF, Zibrowski EM, et al. Rules of engagement: residents’ perceptions of the in-training evaluation process. Acad Med. 2008;83:S97–100.CrossRef Watling CJ, Kenyon CF, Zibrowski EM, et al. Rules of engagement: residents’ perceptions of the in-training evaluation process. Acad Med. 2008;83:S97–100.CrossRef
40.
go back to reference Watling CJ, Lingard L. Toward meaningful evaluation of medical trainees: the influence of participants’ perceptions of the process. Adv Health Sci Educ. 2012;17:183–94.CrossRef Watling CJ, Lingard L. Toward meaningful evaluation of medical trainees: the influence of participants’ perceptions of the process. Adv Health Sci Educ. 2012;17:183–94.CrossRef
41.
go back to reference Isaacson J, Posk L, Litaker D, Halperin A. Resident perception of the evaluation process. J Gen Intern Med. 1995;10:S89.CrossRef Isaacson J, Posk L, Litaker D, Halperin A. Resident perception of the evaluation process. J Gen Intern Med. 1995;10:S89.CrossRef
42.
go back to reference Solomon DJ. Conducting web-based surveys. Pract Assess Res Eval. 2001;7:1–4. Solomon DJ. Conducting web-based surveys. Pract Assess Res Eval. 2001;7:1–4.
Metadata
Title
Students as stakeholders in assessment: how students perceive the value of an assessment
Authors
Michelle Ricci
Christina St-Onge
Jing Xiao
Meredith Young
Publication date
01-12-2018
Publisher
Bohn Stafleu van Loghum
Published in
Perspectives on Medical Education / Issue 6/2018
Print ISSN: 2212-2761
Electronic ISSN: 2212-277X
DOI
https://doi.org/10.1007/s40037-018-0480-3

Other articles of this Issue 6/2018

Perspectives on Medical Education 6/2018 Go to the issue