Top

Implementation Science

Published in:

Open Access 01-12-2018 | Research

Applying modern measurement approaches to constructs relevant to evidence-based practice among Canadian physical and occupational therapists

Authors: Fadi Al Zoubi, Nancy Mayo, Annie Rochette, Aliki Thomas

Published in: Implementation Science | Issue 1/2018

Abstract

Background

Evidence-based practice (EBP) is a complex process. To quantify it, one has to also consider individual and contextual factors using multiple measures. Modern measurement approaches are available to optimize the measurement of complex constructs. This study aimed to develop a robust measurement approach for constructs around EBP including practice, individual (e.g. knowledge, attitudes, confidence, behaviours), and contextual factors (e.g. resources).

Methods

One hundred eighty-one items arising from 5 validated EBP measures were subjected to an item analysis. Nominal group technique was used to arrive at a consensus about the content relevance of each item. Baseline questionnaire responses from a longitudinal study of the evolution of EBP in 128 new graduates of Canadian physical and occupational therapy programmes were analysed. Principles of Rasch Measurement Theory were applied to identify challenges with threshold ordering, item and person fit to the Rasch model, unidimensionality, local independence, and differential item functioning (DIF).

Results

The nominal group technique identified 70/181 items, and modified Delphi approach identified 68 items that fit a formative model (2 related EBP domains: self-use of EBP (9 items) and EBP activities (7 items)) or a reflective model (4 related EBP domains: attitudes towards EBP (17 items), self-efficacy (9 items), knowledge (11 items) and resources (15 items)). Rasch analysis provided a single score for reflective construct. Among attitudes items, 65% (11/17) fit the Rasch model, item difficulties ranged from − 7.51 to logits (least difficult) to + 5.04 logits (most difficult), and person separation index (PSI) = 0.63. Among self-efficacy items, 89% (8/9) fit the Rasch model, item difficulties ranged from − 3.70 to + 4.91, and PSI = 0.80. Among knowledge items, 82% (9/11) fit the Rasch model, item difficulties ranged from − 7.85 to 4.50, and PSI = 0.81. Among resources items, 87% (13/15) fit the Rasch model, item difficulties ranged from − 3.38 to 2.86, and PSI = 0.86. DIF occurred in 2 constructs: attitudes (1 by profession and 2 by language) and knowledge (1 by language and 2 by profession) arising from poor wording in the original version leading to poor translation.

Conclusions

Rasch Measurement Theory was applied to develop a valid and reliable measure of EBP. Further modifications to the items can be done for subsequent waves of the survey.

Available only for authorised users

Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn’t. BMJ. 1996;312(7023):71–2.CrossRef

Feng X, Liu C, Guo Q, Bai Y, Ren Y, Ren B, et al. Research progress in rehabilitation treatment of stroke patients: a bibliometric analysis. Neural Regen Res. 2013;8(15):1423–30. https://doi.org/10.3969/j.issn.1673-5374.2013.15.010.CrossRefPubMedPubMedCentral

Frontera WR. Some reflections on the past, present, and future of physical and rehabilitation medicine (on the occasion of the 30th SOFMER congress). Ann Phys Rehabil Med. 2016;59(2):79–82. https://doi.org/10.1016/j.rehab.2016.02.002.CrossRefPubMed

Kleim JA, Jones TA. Principles of experience-dependent neural plasticity: implications for rehabilitation after brain damage. J Speech Lang Hear Res. 2008;51(1):S225–39. https://doi.org/10.1044/1092-4388(2008/018).CrossRefPubMed

Winstein CJ, Kay DB. Translating the science into practice: shaping rehabilitation practice to enhance recovery after brain damage. Prog Brain Res. 2015;218:331–60. https://doi.org/10.1016/bs.pbr.2015.01.004.CrossRefPubMed

Moore JL, Shikako-Thomas K, Backus D. Knowledge translation in rehabilitation: a shared vision. Pediatr Phys Ther. 2017;29(Suppl 3 Supplement, IV Step 2016 Conference Proceedings Supplement):S64–s72. https://doi.org/10.1097/pep.0000000000000381.CrossRefPubMed

World Health Organisation. Rehabilitation: WHO framework on rehabilitation services: expert meeting. 2017.

Korner-Bitensky N, Desrosiers J, Rochette A. A national survey of occupational therapists’ practices related to participation post-stroke. J Rehabil Med. 2008;40(4):291–7. https://doi.org/10.2340/16501977-0167.CrossRefPubMed

Salls J, Dolhi C, Silverman L, Hansen M. The use of evidence-based practice by occupational therapists. Occup Ther Health Care. 2009;23(2):134–45. https://doi.org/10.1080/07380570902773305.CrossRefPubMed

10.

Petzold A, Korner-Bitensky N, Rochette A, Teasell R, Marshall S, Perrier MJ. Driving poststroke: problem identification, assessment use, and interventions offered by Canadian occupational therapists. Top Stroke Rehabil. 2010;17(5):371–9. https://doi.org/10.1310/tsr1705-371.CrossRefPubMed

11.

Menon-Nair A, Korner-Bitensky N, Ogourtsova T. Occupational therapists’ identification, assessment, and treatment of unilateral spatial neglect during stroke rehabilitation in Canada. Stroke. 2007;38(9):2556–62. https://doi.org/10.1161/strokeaha.107.484857.CrossRefPubMed

12.

Accreditation Council for Canadian Physiotherapy Academic Programs. Essential competency profile for physiotherapists in Canada. 2009. http://www.physiotherapyeducation.ca/Resources/Essential%20Comp%20PT%20Profile%202009.pdf. Accessed 15 April 2018.

13.

Canadian Association of Occupational Therapy. Profile of practice of occupational therapists in Canada. 2012. https://www.caot.ca/document/3653/2012otprofile.pdf. Accessed 15 April 2018.

14.

Godin G, Belanger-Gravel A, Eccles M, Grimshaw J. Healthcare professionals’ intentions and behaviours: a systematic review of studies based on social cognitive theories. Implement Sci. 2008;3:36. https://doi.org/10.1186/1748-5908-3-36.CrossRefPubMedPubMedCentral

15.

Thompson-Leduc P, Clayman ML, Turcotte S, Legare F. Shared decision-making behaviours in health professionals: a systematic review of studies based on the Theory of Planned Behaviour. Health Expect. 2015;18(5):754–74. https://doi.org/10.1111/hex.12176.CrossRefPubMed

16.

Kwasnicka D, Dombrowski SU, White M, Sniehotta F. Theoretical explanations for maintenance of behaviour change: a systematic review of behaviour theories. Health Psychol Rev. 2016;10(3):277–96. https://doi.org/10.1080/17437199.2016.1151372.CrossRefPubMedPubMedCentral

17.

Newhouse RP. Instruments to assess organizational readiness for evidence-based practice. J Nurs Adm. 2010;40(10):404–7. https://doi.org/10.1097/NNA.0b013e3181f2ea08.CrossRefPubMed

18.

Gawlinski A. Evidence-based practice changes: measuring the outcome. AACN Adv Crit Care. 2007;18(3):320–2.PubMed

19.

Shaneyfelt T, Baum KD, Bell D, Feldstein D, Houston TK, Kaatz S, et al. Instruments for evaluating education in evidence-based practice: a systematic review. JAMA. 2006;296(9):1116–27. https://doi.org/10.1001/jama.296.9.1116.CrossRefPubMed

20.

Cadorin L, Bagnasco A, Tolotti A, Pagnucci N, Sasso L. Instruments for measuring meaningful learning in healthcare students: a systematic psychometric review. J Adv Nurs. 2016;72(9):1972–90. https://doi.org/10.1111/jan.12926.CrossRefPubMed

21.

Buchanan H, Siegfried N, Jelsma J. Survey instruments for knowledge, skills, attitudes and behaviour related to evidence-based practice in occupational therapy: a systematic review. Occup Ther Int. 2016;23(2):59–90. https://doi.org/10.1002/oti.1398.CrossRefPubMed

22.

Leung K, Trevena L, Waters D. Systematic review of instruments for measuring nurses’ knowledge, skills and attitudes for evidence-based practice. J Adv Nurs. 2014;70(10):2181–95. https://doi.org/10.1111/jan.12454.CrossRefPubMed

23.

Sick J. Rasch measurement in language education part 5: assumptions and requirements of Rasch measurement. Shiken. 2010;14(2):23–9.

24.

Grimby G, Tennant A, Tesio L. The use of raw scores from ordinal scales: time to end malpractice? J Rehabil Med. 2012;44(2):97–8. https://doi.org/10.2340/16501977-0938.CrossRefPubMed

25.

Andrich D. Rasch models for ordered response categories. Encyclopedia Stat Behav Sci. 2005;4:1698–707. https://doi.org/10.1002/0470013192.bsa541.CrossRef

26.

Traub RE. Classical Test Theory in historical perspective. Educ Meas. 1997;16(4):8–14. https://doi.org/10.1111/j.1745-3992.1997.tb00603.x.CrossRef

27.

Akram W, Hussein MS, Ahmad S, Mamat MN, Ismail NE. Validation of the knowledge, attitude and perceived practice of asthma instrument among community pharmacists using Rasch analysis. Saudi Pharm J. 2015;23(5):499–503. https://doi.org/10.1016/j.jsps.2015.01.011.CrossRefPubMedPubMedCentral

28.

Zoanetti N, Griffin P, Beaves M, Wallace EM. Rasch scaling procedures for informing development of a valid Fetal Surveillance Education Program multiple-choice assessment. BMC Med Educ. 2009;9:20. https://doi.org/10.1186/1472-6920-9-20.CrossRefPubMedPubMedCentral

29.

Bobiak SN, Zyzanski SJ, Ruhe MC, Carter CA, Ragan B, Flocke SA, et al. Measuring practice capacity for change: a tool for guiding quality improvement in primary care settings. Qual Manag Health Care. 2009;18(4):278–84. https://doi.org/10.1097/QMH.0b013e3181bee2f5.CrossRefPubMed

30.

Harris-Haywood S, Goode T, Gao Y, Smith K, Bronheim S, Flocke SA, et al. Psychometric evaluation of a cultural competency assessment instrument for health professionals. Med Care. 2014;52(2):e7–e15. https://doi.org/10.1097/MLR.0b013e31824df149.CrossRefPubMed

31.

Blackman IR, Giles T. Psychometric evaluation of a self-report evidence-based practice tool using Rasch analysis. Worldviews Evid Based Nurs. 2015;12(5):253–64. https://doi.org/10.1111/wvn.12105.CrossRefPubMed

32.

Hasnain M, Gruss V, Keehn M, Peterson E, Valenta AL, Kottorp A. Development and validation of a tool to assess self-efficacy for competence in interprofessional collaborative practice. J Interprof Care. 2017;31(2):255–62. https://doi.org/10.1080/13561820.2016.1249789.CrossRefPubMed

33.

Thomas A, Rochette A, Lapointe J, O’Connor K, Ahmed S, Bussières A, et al. Evolution of evidence-based practice: evaluating the contribution of individual and contextual factors to optimize patient care. Canadian Institutes for Health Research, Project scheme grant; 2016-2022. 2016. http://webapps.cihr-irsc.gc.ca/funding/detail_e?pResearchId=8239783&p_version=CRIS&p_language=E&p_session_id=

34.

Fekete C, Boldt C, Post M, Eriks-Hoogland I, Cieza A, Stucki G. How to measure what matters: development and application of guiding principles to select measurement instruments in an epidemiologic study on functioning. Am J Phys Med Rehabil. 2011;90(11 Suppl 2):S29–38. https://doi.org/10.1097/PHM.0b013e318230fe41.CrossRefPubMed

35.

Geyh S, Cieza A, Kollerits B, Grimby G, Stucki G. Content comparison of health-related quality of life measures used in stroke based on the international classification of functioning, disability and health (ICF): a systematic review. Qual Life Res. 2007;16(5):833–51. https://doi.org/10.1007/s11136-007-9174-8.CrossRefPubMed

36.

Estabrooks CA, Squires JE, Cummings GG, Birdsell JM, Norton PG. Development and assessment of the Alberta Context Tool. BMC Health Serv Res. 2009;9:234. https://doi.org/10.1186/1472-6963-9-234.CrossRefPubMedPubMedCentral

37.

Delbecq AL, Van de Ven AH. A group process model for problem identification and program planning. J Appl Behav Sci. 1971;7(4):466–92. https://doi.org/10.1177/002188637100700404.CrossRef

38.

Fink A, Kosecoff J, Chassin M, Brook RH. Consensus methods: characteristics and guidelines for use. Am J Public Health. 1984;74(9):979–83.CrossRef

39.

McEvoy MP, Williams MT, Olds TS, Lewis LK, Petkov J. Evidence-based practice profiles of physiotherapists transitioning into the workforce: a study of two cohorts. BMC Med Educ. 2011;11:100. https://doi.org/10.1186/1472-6920-11-100.CrossRefPubMedPubMedCentral

40.

McEvoy MP, Williams MT, Olds TS. Development and psychometric testing of a trans-professional evidence-based practice profile questionnaire. Med Teach. 2010;32(9):e373–80. https://doi.org/10.3109/0142159x.2010.494741.CrossRefPubMed

41.

Aarons GA, Glisson C, Hoagwood K, Kelleher K, Landsverk J, Cafri G. Psychometric properties and U.S. national norms of the Evidence-Based Practice Attitude Scale (EBPAS). Psychol Assess. 2010;22(2):356–65. https://doi.org/10.1037/a0019188.CrossRefPubMed

42.

Aarons GA. Mental health provider attitudes toward adoption of evidence-based practice: the Evidence-Based Practice Attitude Scale (EBPAS). Ment Health Serv Res. 2004;6(2):61–74.CrossRef

43.

Aarons GA, McDonald EJ, Sheehan AK, Walrath-Greene CM. Confirmatory factor analysis of the evidence-based practice attitude scale in a geographically diverse sample of community mental health providers. Admin Pol Ment Health. 2007;34(5):465–9. https://doi.org/10.1007/s10488-007-0127-x.CrossRef

44.

Salbach NM, Jaglal SB. Creation and validation of the evidence-based practice confidence scale for health care professionals. J Eval Clin Pract. 2011;17(4):794–800. https://doi.org/10.1111/j.1365-2753.2010.01478.x.CrossRefPubMed

45.

Salbach NM, Jaglal SB, Williams JI. Reliability and validity of the evidence-based practice confidence (EPIC) scale. J Contin Educ Heal Prof. 2013;33(1):33–40. https://doi.org/10.1002/chp.21164.CrossRef

46.

Clyde JH, Brooks D, Cameron JI, Salbach NM. Validation of the evidence-based practice confidence (EPIC) scale with occupational therapists. Am J Occup Ther. 2016;70(2):7002280010p1–9. https://doi.org/10.5014/ajot.2016.017061.CrossRefPubMed

47.

Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16(3):297–334. https://doi.org/10.1007/bf02310555.CrossRef

48.

Squires JE, Hayduk L, Hutchinson AM, Mallick R, Norton PG, Cummings GG, et al. Reliability and validity of the Alberta Context Tool (ACT) with professional nurses: findings from a multi-study analysis. PloS one. 2015;10(6):e0127405. https://doi.org/10.1371/journal.pone.0127405.CrossRefPubMedPubMedCentral

49.

Lynn MR. Determination and quantification of content validity. Nurs Res. 1986;35(6):382–5.CrossRef

50.

Beaton DE, Bombardier C, Guillemin F, Ferraz MB. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine (Phila Pa 1976). 2000;25(24):3186–91.CrossRef

51.

Gibbons CJ, Mills RJ, Thornton EW, Ealing J, Mitchell JD, Shaw PJ, et al. Rasch analysis of the hospital anxiety and depression scale (HADS) for use in motor neurone disease. Health Qual Life Outcomes. 2011;9:82. https://doi.org/10.1186/1477-7525-9-82.CrossRefPubMedPubMedCentral

52.

Tennant A, Conaghan PG. The Rasch measurement model in rheumatology: what is it and why use it? When should it be applied, and what should one look for in a Rasch paper? Arthritis Rheum. 2007;57(8):1358–62. https://doi.org/10.1002/art.23108.CrossRefPubMed

53.

Bond TG, Fox CM. Applying the Rasch model: fundamental measurement in the human sciences. Third ed. New York: Routledge; 2015.

54.

Linacre JM. Sample size and item calibration [or person measure] stability. Rasch measurement transactions; 1994. http://www.rasch.org/rmt/rmt74m.htm. Accessed 15 May 2018.

55.

Andrich D, Sheridan B, Luo G. Rasch unidimensional measurement models (RUMM) 2030. Perth: RUMM Pty Ltd; 2008.

56.

Statistical Analysis System Institute Inc. SAS® 9.4 Software. 2014. http://support.sas.com/software/94/index.html. 15 Mar 2018.

57.

Morrison T, Robertson L. New graduates’ experience of evidence-based practice: an action research study. Br J Occup Ther. 2016;79(1):42–8. https://doi.org/10.1177/0308022615591019.CrossRef

58.

Powell CA, Case-Smith J. Information literacy skills of occupational therapy graduates: promoting evidence-based practice in the MOT curriculum. Med Ref Serv Q. 2010;29(4):363–80. https://doi.org/10.1080/02763869.2010.518923.CrossRefPubMed

59.

Thomas A, Law M. Research utilization and evidence-based practice in occupational therapy: a scoping study. Am J Occup Ther. 2013;67(4):e55–65. https://doi.org/10.5014/ajot.2013.006395.CrossRefPubMed

60.

Wilkinson SA, Hinchliffe F, Hough J, Chang A. Baseline evidence-based practice use, knowledge, and attitudes of allied health professionals: a survey to inform staff training and organisational change. J Allied Health. 2012;41(4):177–84.PubMed

61.

Brown T, Tseng MH, Casey J, McDonald R, Lyons C. Research knowledge, attitudes, and practices of pediatric occupational therapists in Australia, the United Kingdom, and Taiwan. J Allied Health. 2010;39(2):88–94.PubMed

62.

Mroz TM, Pitonyak JS, Fogelberg D, Leland NE. Client centeredness and health reform: key issues for occupational therapy. Am J Occup Ther. 2015;69(5):6905090010p1–8. https://doi.org/10.5014/ajot.2015.695001.CrossRefPubMedPubMedCentral

63.

Alsop A. Evidence-based practice and continuing professional development. Br J Occup Ther. 1997;60(11):503–8. https://doi.org/10.1177/030802269706001112.CrossRef

64.

Rogers WT, Lin J, Rinaldi CM. Validity of the simultaneous approach to the development of equivalent achievement tests in English and French. Appl Meas Educ. 2010;24(1):39–70. https://doi.org/10.1080/08957347.2011.532416.CrossRef

65.

Kwakkenbos L, Willems LM, Baron M, Hudson M, Cella D, van den Ende CHM, et al. The comparability of English, French and Dutch scores on the Functional Assessment of Chronic Illness Therapy-Fatigue (FACIT-F): an assessment of differential item functioning in patients with systemic sclerosis. PLoS One. 2014;9(3):e91979. https://doi.org/10.1371/journal.pone.0091979.CrossRefPubMedPubMedCentral

66.

Delisle VC, Kwakkenbos L, Hudson M, Baron M, Thombs BD. An assessment of the measurement equivalence of English and French versions of the Center for Epidemiologic Studies Depression (CES-D) Scale in systemic sclerosis. PLoS One. 2014;9(7):e102897. https://doi.org/10.1371/journal.pone.0102897.CrossRefPubMedPubMedCentral

67.

Arthurs E, Steele RJ, Hudson M, Baron M, Thombs BD. Are scores on English and French versions of the PHQ-9 comparable? An assessment of differential item functioning. PLoS One. 2012;7(12):e52028. https://doi.org/10.1371/journal.pone.0052028.CrossRefPubMedPubMedCentral

68.

Askari S, Fellows L, Brouillette MJ, Moriello C, Duracinsky M, Mayo NE. Development of an item pool reflecting cognitive concerns expressed by people with HIV. Am J Occup Ther. 2018;72(2):7202205070p1–9. https://doi.org/10.5014/ajot.2018.023945.CrossRefPubMed

69.

Stapleton T, McBrearty C. Use of standardised assessments and outcome measures among a sample of Irish occupational therapists working with adults with physical disabilities. Br J Occup Ther. 2009;72(2):55–64. https://doi.org/10.1177/030802260907200203.CrossRef

70.

Bowman J. Challenges to measuring outcomes in occupational therapy: a qualitative focus group study. Br J Occup Ther. 2006;69(10):464–72. https://doi.org/10.1177/030802260606901005.CrossRef

71.

Atwal A, McIntyre A, Craik C, Hunt J. Occupational therapists’ perceptions of predischarge home assessments with older adults in acute care. Br J Occup Ther. 2008;71(2):52–8. https://doi.org/10.1177/030802260807100203.CrossRef

72.

Fowler F. Survey research methods. 4th ed. Thousand Oaks; 2009. https://doi.org/10.4135/9781452230184.

73.

Moullin JC, Ehrhart MG, Aarons GA. Development and testing of the Measure of Innovation-Specific Implementation Intentions (MISII) using Rasch measurement theory. Implement Sci. 2018;13(1):89. https://doi.org/10.1186/s13012-018-0782-1.CrossRefPubMedPubMedCentral

74.

Cane J, O’Connor D, Michie S. Validation of the theoretical domains framework for use in behaviour change and implementation research. Implement Sci. 2012;7:37. https://doi.org/10.1186/748-5908-7-37.CrossRefPubMedPubMedCentral

75.

Ajzen I. The theory of planned behavior. Organ Behav Hu Dec Processes. 1991;50(2):179–211 doi:https://doi.org/10.1016/0749-5978(91)90020-T.CrossRef

76.

Stevenson K, Lewis M, Hay E. Do physiotherapists’ attitudes towards evidence-based practice change as a result of an evidence-based educational programme? J Eval Clin Pract. 2004;10(2):207–17. https://doi.org/10.1111/j.1365-2753.2003.00479.x.CrossRefPubMed

77.

Stevenson K, Lewis M, Hay E. Does physiotherapy management of low back pain change as a result of an evidence-based educational programme? J Eval Clin Pract. 2006;12(3):365–75. https://doi.org/10.1111/j.1365-2753.2006.00565.x.CrossRefPubMed

78.

Flores-Mateo G, Argimon JM. Evidence based practice in postgraduate healthcare education: a systematic review. BMC Health Serv Res. 2007;7:119. https://doi.org/10.1186/1472-6963-7-119.CrossRefPubMedPubMedCentral

Title: Applying modern measurement approaches to constructs relevant to evidence-based practice among Canadian physical and occupational therapists
Authors: Fadi Al Zoubi
Nancy Mayo
Annie Rochette
Aliki Thomas
Publication date: 01-12-2018
Publisher: BioMed Central
Published in: Implementation Science / Issue 1/2018
Electronic ISSN: 1748-5908
DOI: https://doi.org/10.1186/s13012-018-0844-4

Keynote webinar | Spotlight on sleep in brain health

Springer Medicine

Applying modern measurement approaches to constructs relevant to evidence-based practice among Canadian physical and occupational therapists

Abstract

Background

Methods

Results

Conclusions

Keynote webinar | Spotlight on sleep in brain health

Springer Medicine

Abstract

Background

Methods

Results

Conclusions

Please log in to get access to this content

Other articles of this Issue 1/2018

Enhancing the uptake of systematic reviews of effects: what is the best format for health care managers and policy-makers? A mixed-methods study

Correction to: Updated clinical guidelines experience major reporting limitations

Implementation science in resource-poor countries and communities

Implementation strategies to improve cervical cancer prevention in sub-Saharan Africa: a systematic review

Why the uptake of eRehabilitation programs in stroke care is so difficult—a focus group study in the Netherlands

Using Normalization Process Theory in feasibility studies and process evaluations of complex healthcare interventions: a systematic review