Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

Keynote webinar | Spotlight on medication adherence

LIVE: Thursday 27th June 2024, 18:00-19:30 (CEST)
Join our expert panel to discover why you need to understand the drivers of non-adherence in your patients, and how you can optimize medication adherence in your clinics to drastically improve patient outcomes.
ADVANCED SEARCH
Log in
Top
BMC Medical Informatics and Decision Making
Published in: BMC Medical Informatics and Decision Making 1/2021

Open Access 01-12-2021 | Telemedicine | Research article

The most used questionnaires for evaluating telemedicine services

Authors: Sadrieh Hajesmaeel-Gohari, Kambiz Bahaadinbeigy

Published in: BMC Medical Informatics and Decision Making | Issue 1/2021

Login to get access

Abstract

Background

Questionnaires are commonly used tools in telemedicine services that can help to evaluate different aspects. Selecting the ideal questionnaire for this purpose may be challenging for researchers. This study aims to review which questionnaires are used to evaluate telemedicine services in the studies, which are most common, and what aspects of telemedicine evaluation do they capture.

Methods

The PubMed database was searched in August 2020 to retrieve articles. Data extracted from the final list of articles included author/year of publication, journal of publication, type of evaluation, and evaluation questionnaire. Data were analyzed using descriptive statistics.

Results

Fifty-three articles were included in this study. The questionnaire was used for evaluating the satisfaction (49%), usability (34%), acceptance (11.5%), and implementation (2%) of telemedicine services. Among telemedicine specific questionnaires, Telehealth Usability Questionnaire (TUQ) (19%), Telemedicine Satisfaction Questionnaire (TSQ) (13%), and Service User Technology Acceptability Questionnaire (SUTAQ) (5.5%), were respectively most frequently used in the collected articles. Other most used questionnaires generally used for evaluating the users’ satisfaction, usability, and acceptance of technology were Client Satisfaction Questionnaire (CSQ) (5.5%), Questionnaire for User Interaction Satisfaction (QUIS) (5.5%), System Usability Scale (SUS) (5.5%), Patient Satisfaction Questionnaire (PSQ) (5.5%), and Technology Acceptance Model (TAM) (3.5%) respectively.

Conclusion

Employing specifically designed questionnaires or designing a new questionnaire with fewer questions and more comprehensiveness in terms of the issues studied provides a better evaluation. Attention to user needs, end-user acceptance, and implementation processes, along with users' satisfaction and usability evaluation, may optimize telemedicine efforts in the future.
Literature
1.
2.
3.
4.
Ammenwerth E, Brender J, Nykänen P, Prokosch H-U, Rigby M, Talmon J. Visions and strategies to improve evaluation of health information systems: reflections and lessons based on the HIS-EVAL workshop in Innsbruck. Int J Med Informat. 2004;73(6):479–91.CrossRef
5.
McNair JB. Handbook of evaluation methods for health informatics. Amsterdam: Elsevier; 2006.
6.
AlDossary S, Martin-Khan MG, Bradford NK, Smith ACJI. A systematic review of the methodologies used to evaluate telemedicine service initiatives in hospital facilities. Int J Med Informat. 2017;97:171–94.CrossRef
7.
Klaassen B, van Beijnum BJ, Hermens HJ. Usability in telemedicine systems—a literature survey. Int J Med Inf. 2016;93:57–69.CrossRef
8.
Kidholm K, Clemensen J, Caffery LJ, Smith AC. The Model for Assessment of Telemedicine (MAST): a scoping review of empirical studies. J Telemed Telecare. 2017;23(9):803–13.PubMedCrossRef
9.
Bahaadinbeigy K, Yogesan K, Wootton R. MEDLINE versus EMBASE and CINAHL for telemedicine searches. Telemed J E-health. 2010;16(8):916–9.PubMedCrossRef
10.
Layfield E, Triantafillou V, Prasad A, Deng J, Shanti RM, Newman JG et al. Telemedicine for head and neck ambulatory visits during COVID‐19: evaluating usability and patient satisfaction. Head Neck. 2020:1–9.
11.
Mostafa PI, Hegazy AA. Dermatological consultations in the COVID-19 era: Is teledermatology the key to social distancing? An Egyptian experience. J Dermatol Treat. 2020:1–6.
12.
Whitehouse CR, Long JA, Maloney LM, Daniels K, Horowitz DA, Bowles KH. Feasibility of diabetes self-management telehealth education for older adults during transitions in care. Res Gerontol Nurs. 2020;13(3):138–45.PubMedCrossRef
13.
Vaughan EM, Naik AD, Lewis CM, Foreyt JP, Samson SL, Hyman DJ. Telemedicine training and support for community health workers: improving knowledge of diabetes. Telemed J E-health. 2020;26(2):244–50.PubMedPubMedCentralCrossRef
14.
Cheng O, Law N-H, Tulk J, Hunter M. Utilization of telemedicine in addressing musculoskeletal care gap in long-term care patients. JAAOS Global Res Rev. 2020;4(4).
15.
16.
Serwe KM, Hersch GI, Pancheri KJ. Feasibility of using telehealth to deliver the “Powerful Tools for Caregivers” program. Int J Telerehabil. 2017;9(1):15.PubMedPubMedCentralCrossRef
17.
Yu DX, Parmanto B, Dicianno BE, Watzlaf VJ, Seelman KD. Accessibility needs and challenges of a mHealth system for patients with dexterity impairments. Disabil Rehabil Assist Technol. 2017;12(1):56–64.PubMedCrossRef
18.
19.
Schutte J, Gales S, Filippone A, Saptono A, Parmanto B, McCue M. Evaluation of a telerehabilitation system for community-based rehabilitation. Int J Telerehabil. 2012;4(1):15.PubMedPubMedCentral
20.
Lin CH, Tseng WP, Wu JL, Tay J, Cheng MT, Ong HN, et al. A double triage and telemedicine protocol to optimize infection control in an emergency department in Taiwan during the COVID-19 pandemic: retrospective feasibility study. J Med Internet Res. 2020;22(6):e20586.PubMedPubMedCentralCrossRef
21.
Talal AH, McLeod A, Andrews P, Nieves-McGrath H, Chen Y, Reynolds A, et al. Patient reaction to telemedicine for clinical management of hepatitis C virus integrated into an opioid treatment program. Telemed J E-Health. 2019;25(9):791–801.PubMedCrossRef
22.
Le LB, Rahal HK, Viramontes MR, Meneses KG, Dong TS, Saab S. Patient satisfaction and healthcare utilization using telemedicine in liver transplant recipients. Dig Dis Sci. 2019;64(5):1150–7.PubMedCrossRef
23.
Losiouk E, Lanzola G, Del Favero S, Boscari F, Messori M, Rabbone I, et al. Parental evaluation of a telemonitoring service for children with type 1 diabetes. J Telemed Telecare. 2018;24(3):230–7.PubMedCrossRef
24.
Segura-Sampedro JJ, Rivero-Belenchón I, Pino-Díaz V, Rodríguez Sánchez MC, Pareja-Ciuró F, Padillo-Ruiz J, et al. Feasibility and safety of surgical wound remote follow-up by smart phone in appendectomy: a pilot study. Ann Med Surgery. 2012;2017(21):58–62.
25.
26.
Yip MP, Chang AM, Chan J, MacKenzie AE. Development of the telemedicine satisfaction questionnaire to evaluate patient satisfaction with telemedicine: a preliminary study. J Telemed Telecare. 2003;9(1):46–50.PubMedCrossRef
27.
Torbjørnsen A, Småstuen MC, Jenum AK, Årsand E, Ribu L. The service user technology acceptability questionnaire: psychometric evaluation of the norwegian version. JMIR Human Factors. 2018;5(4):e10255.PubMedPubMedCentralCrossRef
28.
Dario C, Luisotto E, Dal Pozzo E, Mancin S, Aletras V, Newman S, et al. Assessment of patients’ perception of telemedicine services using the service user technology acceptability questionnaire. Int J Integrated Care. 2016;16(2):13.CrossRef
29.
Hirani SP, Rixon L, Beynon M, Cartwright M, Cleanthous S, Selva A, et al. Quantifying beliefs regarding telehealth: development of the whole systems demonstrator service user technology acceptability questionnaire. J Telemed Telecare. 2017;23(4):460–9.PubMedCrossRef
30.
Christensen LF, Gildberg FA, Sibbersen C, Skjoeth MM, Nielsen CT, Hansen JP. Videoconferences and treatment of depression: satisfaction score correlated with number of sessions attended but not with age. Telemed J E-Health. 2020;26(7):898–904.PubMedCrossRef
31.
Christensen LF, Gildberg FA, Sibbersen C, Skjoeth MM, Nielsen CT, Hansen JP. Disagreement in satisfaction between patients and providers in the use of videoconferences by depressed adults. Telemed J E-health. 2020;26(5):614–20.PubMedCrossRef
32.
Fields BG, Behari PP, McCloskey S, True G, Richardson D, Thomasson A, et al. Remote ambulatory management of veterans with obstructive sleep apnea. Sleep. 2016;39(3):501–9.PubMedPubMedCentralCrossRef
33.
Safdari R, Langarizadeh M, Ramezani A, Khodaveisi T, Nejad AF. Development of a store-and-forward telescreening system of diabetic retinopathy: lessons learned from Iran. J Diabetes Metabolic Disorders. 2018;17(1):31–6.CrossRef
34.
Hosseini F, Ayatollahi H, Salehi SH, Jafar K. Teleburn: designing a telemedicine application to improve burn treatment. Open Med Informat J. 2018;12:33–41.CrossRef
35.
Alanzi T, Istepanian R, Philip N. Design and usability evaluation of social mobile diabetes management system in the Gulf Region. JMIR Res Protocols. 2016;5(3):e93.CrossRef
36.
Smaradottir B, Gerdes M, Martinez S, Fensli R. The EU-project United4Health: user-centred design of an information system for a Norwegian telemedicine service. J Telemed Telecare. 2016;22(7):422–9.PubMedCrossRef
37.
Ligons FM, Mello-Thoms C, Handler SM, Romagnoli KM, Hochheiser H. Assessing the impact of cognitive impairment on the usability of an electronic medication delivery unit in an assisted living population. Int J Med Informat. 2014;83(11):841–8.CrossRef
38.
Lacerda TC, von Wangenheim CG, von Wangenheim A, Giuliano I. Does the use of structured reporting improve usability? A comparative evaluation of the usability of two approaches for findings reporting in a large-scale telecardiology context. J Biomed Inform. 2014;52:222–30.PubMedCrossRef
39.
McFarland LV, Raugi GJ, Reiber GE. Primary care provider and imaging technician satisfaction with a teledermatology project in rural Veterans Health Administration clinics. Telemed J E-health. 2013;19(11):815–25.PubMedCrossRef
40.
Wallace P, Haines A, Harrison R, Barber JA, Thompson S, Roberts J, et al. Design and performance of a multi-centre randomised controlled trial and economic evaluation of joint tele-consultations [ISRCTN54264250]. BMC Family Practice. 2002;3:1.PubMedPubMedCentralCrossRef
41.
McGloin H, O’Connell D, Glacken M, Mc Sharry P, Healy D, Winters-O’Donnell L, et al. Patient empowerment using electronic telemonitoring with telephone support in the transition to insulin therapy in adults with type 2 diabetes: observational, pre-post, mixed methods study. J Med Internet Res. 2020;22(5):e16161.PubMedPubMedCentralCrossRef
42.
Bakken S, Grullon-Figueroa L, Izquierdo R, Lee N-J, Morin P, Palmas W, et al. Development, validation, and use of english and spanish versions of the telemedicine satisfaction and usefulness questionnaire. J Am Med Inform Assoc. 2006;13(6):660.PubMedPubMedCentralCrossRef
43.
Hatton J, Chandra R, Lucius D, Ciuchta E. Patient satisfaction of pharmacist-provided care via clinical video teleconferencing. J Pharm Pract. 2018;31(5):429–33.PubMedCrossRef
44.
Agha Z, Schapira RM, Laud PW, McNutt G, Roter DL. Patient satisfaction with physician-patient communication during telemedicine. Telemed J E-health. 2009;15(9):830–9.PubMedCrossRef
45.
Kwon S, Kim J, Kang S, Lee Y, Baek H, Park K. CardioGuard: a brassiere-based reliable ECG monitoring sensor system for supporting daily smartphone healthcare applications. Telemed J E-Health. 2014;20(12):1093–102.PubMedPubMedCentralCrossRef
46.
Agnisarman SO, Chalil Madathil K, Smith K, Ashok A, Welch B, McElligott JT. Lessons learned from the usability assessment of home-based telemedicine systems. Appl Ergonom. 2017;58:424–34.CrossRef
47.
Parra C, Jódar-Sánchez F, Jiménez-Hernández MD, Vigil E, Palomino-García A, Moniche-Álvarez F, et al. Development, implementation, and evaluation of a telemedicine service for the treatment of acute stroke patients: TeleStroke. Interact J Med Res. 2012;1(2):e15.PubMedPubMedCentralCrossRef
48.
Mehra S, van den Helder J, Visser B, Engelbert RH, Weijs PJ, Kröse BJ. Evaluation of a blended physical activity intervention for older adults: mixed methods study. J Med Internet Res. 2020;22(7):e16380.PubMedPubMedCentralCrossRef
49.
Leppert F, Siebermair J, Wesemann U, Martens E, Sattler SM, Scholz S, et al. The INFluence of remote monitoring on anxiety/depression, quality of life, and device acceptance in ICD patients: a prospective, randomized, controlled, single-center trial. Clin Res Cardiol. 2020.
50.
van der Meij E, Huirne JA, Ten Cate AD, Stockmann HB, Scholten PC, Davids PH, et al. A perioperative ehealth program to enhance postoperative recovery after abdominal surgery: process evaluation of a randomized controlled trial. J Med Internet Res. 2018;20(1):e1.PubMedPubMedCentralCrossRef
51.
Host BK, Turner AW, Muir J. Real-time teleophthalmology video consultation: an analysis of patient satisfaction in rural Western Australia. Clin Exper Optometry. 2018;101(1):129–34.CrossRef
52.
D’Hooghe M, Van Gassen G, Kos D, Bouquiaux O, Cambron M, Decoo D, et al. Improving fatigue in multiple sclerosis by smartphone-supported energy management: the MS TeleCoach feasibility study. Multiple Sclerosis Related Disorders. 2018;22:90–6.PubMedCrossRef
53.
Ammenwerth E, Modre-Osprian R, Fetz B, Gstrein S, Krestan S, Dörler J, et al. HerzMobil, an integrated and collaborative telemonitoring-based disease management program for patients with heart failure: a feasibility study paving the way to routine care. JMIR Cardio. 2018;2(1):e11.PubMedPubMedCentralCrossRef
54.
55.
Roberts S, Spain B, Hicks C, London J, Tay S. Telemedicine in the Northern Territory: an assessment of patient perceptions in the preoperative anaesthetic clinic. Aust J Rural Health. 2015;23(3):136–41.PubMedCrossRef
56.
Poulsen KA, Millen CM, Lakshman UI, Buttner PG, Roberts LJ. Satisfaction with rural rheumatology telemedicine service. Int J Rheumatic Diseases. 2015;18(3):304–14.CrossRef
57.
Vélez O, Okyere PB, Kanter AS, Bakken S. A usability study of a mobile health application for rural Ghanaian midwives. J Midwifery Women’s Health. 2014;59(2):184–91.CrossRef
58.
Penteado SP, Bento RF, Battistella LR, Silva SM, Sooful P. Use of the satisfaction with amplification in daily life questionnaire to assess patient satisfaction following remote hearing aid adjustments (telefitting). JMIR Med Informat. 2014;2(2):e18.CrossRef
59.
Scalvini S, Tridico C, Glisenti F, Giordano A, Pirini S, Peduzzi P, et al. The SUMMA project: a feasibility study on telemedicine in selected Italian areas. Telemed J E-health. 2009;15(3):261–9.PubMedCrossRef
60.
Kim HM, Lowery JC, Hamill JB, Wilkins EG. Patient attitudes toward a web-based system for monitoring chronic wounds. Telemed J E-Health. 2004;10(Suppl 2):S26-34.CrossRef
61.
Eminovic N, Wyatt JC, Tarpey AM, Murray G, Ingrams GJ. First evaluation of the NHS direct online clinical enquiry service: a nurse-led web chat triage service for the public. J Med Internet Res. 2004;6(2):e17.PubMedPubMedCentralCrossRef
62.
van Rosmalen-Nooijens K, Lo Fo Wong S, Prins J, Lagro-Janssen T. Young people, adult worries: randomized controlled trial and feasibility study of the internet-based self-support method “feel the vibe” for adolescents and young adults exposed to family violence. J Med Internet Res. 2017;19(6):e204.PubMedPubMedCentralCrossRef
63.
Morgan DG, Kosteniuk J, Stewart N, O’Connell ME, Karunanayake C, Beever R. The telehealth satisfaction scale: reliability, validity, and satisfaction with telehealth in a rural memory clinic population. Telemed J E-health. 2014;20(11):997–1003.PubMedCrossRef
64.
Lund AM. Measuring usability with the USE questionnaire. Usability Interface. 2001;8(2):3–6.
65.
Attkisson CC, Zwick R. The client satisfaction questionnaire: psychometric properties and correlations with service utilization and psychotherapy outcome. Eval Program Plan. 1982;5(3):233–7.CrossRef
66.
Burns JL, Serber ER, Keim S, Sears SF, et al. Measuring patient acceptance of implantable cardiac device therapy: initial psychometric investigation of the Florida Patient Acceptance Survey. J Cardiovasc Electrophysiol. 2005;16(4):384–90.PubMedCrossRef
67.
Marshall GN, Hays RD. The patient satisfaction questionnaire short-form (PSQ-18). RAND; 1994.
68.
Linnan L, Steckler A. Process evaluation for public health interventions and research. 2002.
69.
Chin J, Diehl V, Norman KL. Development of an Instrument Measuring User Satisfaction of the Human-Computer Interface. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1988:213–8.
70.
Fatehi F, Martin-Khan M, Smith AC, Russell AW, Gray LCJDT. Therapeutics Patient satisfaction with video teleconsultation in a virtual diabetes outreach clinic. Diabetes Technol Ther. 2015;17(1):43–8.PubMedCrossRef
71.
Wessel R. KH, De Witte L. D-Quest meetinstrument voor de tevredenheid over een hulpmiddel verstrekking. Hoensbroek. 2000.
72.
Delone WH, McLean ERJJ. The DeLone and McLean model of information systems success: a ten-year update. J Manag Inf Syst. 2003;19(4):9–30.CrossRef
73.
Elling S, Lentz L, De Jong M, editors. Website evaluation questionnaire: development of a research-based tool for evaluating informational websites. In: International conference on electronic government; 2007. Springer.
74.
Wang Y-SJI. Management. Assessment of learner satisfaction with asynchronous electronic learning systems. Inf Manag. 2003;41(1):75–86.CrossRef
75.
Lewis JR, JIJoHCI. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int J Human Comput Interact. 1995;7(1):57–78.CrossRef
76.
Hart SG, Staveland LE. Development of NASA-TLX (task load index): results of empirical and theoretical research. Adv Psychol. 1988;52:139–83.CrossRef
77.
Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. 1989;13(3):319–40.CrossRef
78.
Brooke J. SUS: A “Quick and Dirty” Usability Scale. Milton Park: Taylor & Francis Group; 1986.
79.
Loane M, Bloomer S, Corbett R, Eedy D, Gore H, Mathews C, et al. Patient satisfaction with realtime teledermatology in Northern Ireland. J Telemed Telecare. 1998;4(1):36–40.PubMedCrossRef
80.
Sabesan S, Simcox K, Marr I. Medical oncology clinics through videoconferencing: an acceptable telehealth model for rural patients and health workers. Internal Med J. 2012;42(7):780–5.CrossRef
81.
Yen P-Y, Wantland D, Bakken S, editors. Development of a customizable health IT usability evaluation scale. AMIA Annual Symposium Proceedings; 2010: American Medical Informatics Association.
82.
Cox RM, Alexander GC. Measuring satisfaction with amplification in daily life: the SADL scale. Ear Hear. 1999;20(4):306–20.PubMedCrossRef
83.
Sauro J, Lewis JR. Quantifying the User Experience: Practical Statistics for User Research. Amsterdam: Elsevier; 2012.
84.
Ware JE Jr, Snyder MK, Wright WR, Davies AR. Defining and measuring patient satisfaction with medical care. Evaluat Program Plan. 1983;6(3–4):247–63.CrossRef
85.
Lewis JR. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int J Human Comput Interact. 1995;7(1):57–78.CrossRef
86.
Bertini G. Il learning audit nella sanità. Guida ai processi di miglioramento autogestiti: FrancoAngeli; 2003.
87.
Brick JE, Bashshur RL, Brick JF, D’alessandri RMJTJ, . Public knowledge, perception, and expressed choice of telemedicine in rural West Virginia. Telemed J. 1997;3(2):159–71.PubMedCrossRef
88.
Demiris G, et al. A questionnaire for the assessment of patients’ impressions of the risks and benefits of home telecare. J Telemed Telecare. 2000;6(5):278–84.PubMedCrossRef
89.
Davis R, Gardner J, Schnall R. A review of usability evaluation methods and their use for testing eHealth HIV interventions. Current HIV/AIDS Reports. 2020:1–16.
Metadata
Title
The most used questionnaires for evaluating telemedicine services
Authors
Sadrieh Hajesmaeel-Gohari
Kambiz Bahaadinbeigy
Publication date
01-12-2021
Publisher
BioMed Central
Keyword
Telemedicine
Published in
BMC Medical Informatics and Decision Making / Issue 1/2021
Electronic ISSN: 1472-6947
DOI
https://doi.org/10.1186/s12911-021-01407-y

Other articles of this Issue 1/2021

BMC Medical Informatics and Decision Making 1/2021 Go to the issue

Research article

Generating real-world evidence from unstructured clinical notes to examine clinical utility of genetic tests: use case in BRCAness

Research

A home-based pulmonary rehabilitation mHealth system to enhance the exercise capacity of patients with COPD: development and evaluation

Technical advance

Use of electronic pharmacy transaction data and website development to assess antibiotic use in nursing homes

Research article

Use of a quantitative data report in a hypothetical decision scenario for health policymaking: a computer-assisted laboratory study

Research

Determining the effect of tele-rehabilitation on patients with stutter using the goal attainment scaling (GAS)

Research article

An OMOP-CDM based pharmacovigilance data-processing pipeline (PDP) providing active surveillance for ADR signal detection from real-world data sources