Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

Keynote webinar | Spotlight on medication adherence

LIVE: Thursday 27th June 2024, 18:00-19:30 (CEST)
Join our expert panel to discover why you need to understand the drivers of non-adherence in your patients, and how you can optimize medication adherence in your clinics to drastically improve patient outcomes.
ADVANCED SEARCH
Log in
Top
Trials
Published in: Trials 1/2023

Open Access 01-12-2023 | Research

Rating versus ranking in a Delphi survey: a randomized controlled trial

Authors: Claudio Del Grande, Janusz Kaczorowski

Published in: Trials | Issue 1/2023

Login to get access

Abstract

Background

The Delphi technique has steeply grown in popularity in health research as a structured approach to group communication process. Rating and ranking are two different procedures commonly used to quantify participants’ opinions in Delphi surveys. We explored the influence of using a rating or ranking approach on item prioritization (main outcome), questionnaire completion time, and evaluation of task difficulty in a Delphi survey aimed at identifying priorities for the organization of primary cardiovascular care.

Methods

A randomized controlled parallel group trial was embedded in a three-round online Delphi survey. After an “open” first round, primary care patients, trained patient partners, and primary care clinicians from seven primary care practices were allocated 1:1 to a rating or ranking assessment group for the remainder of the study by stratified permuted block randomization, with strata based on participants’ gender and status. Agreement on item prioritization between the experimental groups was measured by calculating Krippendorff’s alpha reliability coefficient on the aggregate rank order of items in each group after the final round. Self-reported ease or difficulty with the assessment task was measured with the Single Ease Question.

Results

Thirty-six panelists (13 clinic patients, 7 patient partners, 16 clinicians; 60% females) were randomized to the rating (n = 18) or ranking (n = 18) group, with 30 (83%) completing all rounds. Both groups identified the same highest priorities from a set of 41 items, but significant discrepancies were found as early as the seventh top item. There was moderately strong agreement between the priority ordering of top items common to both groups (Krippendorff’s alpha = 0.811, 95% CI = 0.669–0.920). A 9-min mean difference to complete the third-round questionnaire in favor of the rating group failed to achieve statistical significance (p = 0.053). Ranking was perceived as more difficult (p < 0.001).

Conclusions

A rating or ranking procedure led to modestly similar item prioritization in a Delphi survey, but ranking was more difficult. This study should be replicated with a larger number of participants and with variations in the ranking and rating procedures.

Trial registration

Not applicable.
Appendix
Available only for authorised users
Literature
1.
Dalkey N, Helmer O. An experimental application of the Delphi method to the use of experts. Manage Sci. 1963;9:458–67.
2.
Dalkey N. An experimental study of group opinion. Futures. 1969;1:408–26.
3.
4.
Linstone HA, Turoff M, editors. The Delphi method: techniques and applications. Reading, MA: Addison-Wesley Publishing Company; 1975.
5.
Vernon W. The Delphi technique: a review. Int J Ther Rehab. 2009;16:69–76.
6.
Goodman CM. The Delphi technique: a critique. J Adv Nurs. 1987;12:729–34.PubMed
7.
Crisp J, Pelletier D, Duffield C, Adams A, Nagy S. The Delphi method? Nurs Res. 1997;46:116–8.PubMed
8.
Keeney S, Hasson F, McKenna HP. A critical review of the Delphi technique as a research methodology for nursing. Int J Nurs Stud. 2001;38:195–200.PubMed
9.
Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS ONE. 2011;6:e20476.PubMedPubMedCentral
10.
Hasson F, Keeney S. Enhancing rigour in the Delphi technique research. Technol Forecast Soc Change. 2011;78:1695–704.
11.
Paré G, Cameron A-F, Poba-Nzaou P, Templier M. A systematic assessment of rigor in information systems ranking-type Delphi studies. Inf Manag. 2013;50:207–17.
12.
Diamond IR, Grant RC, Feldman BM, Pencharz PB, Ling SC, Moore AM, et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol. 2014;67:401–9.PubMed
13.
Jünger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review. Palliat Med. 2017;31:684–706.PubMed
14.
Grant S, Booth M, Khodyakov D. Lack of preregistered analysis plans allows unacceptable data mining for and selective reporting of consensus in Delphi studies. J Clin Epidemiol. 2018;99:96–105.PubMed
15.
Belton I, MacDonald A, Wright G, Hamlin I. Improving the practical application of the Delphi method in group-based judgment: a six-step prescription for a well-founded and defensible process. Technol Forecast Soc Change. 2019;147:72–82.
16.
Humphrey-Murto S, de Wit M. The Delphi method-more research please. J Clin Epidemiol. 2019;106:136–9.PubMed
17.
Brookes ST, Macefield RC, Williamson PR, McNair AG, Potter S, Blencowe NS, et al. Three nested randomized controlled trials of peer-only or multiple stakeholder group feedback within Delphi surveys during core outcome and information set development. Trials. 2016;17:409.PubMedPubMedCentral
18.
MacLennan S, Kirkham J, Lam TBL, Williamson PR. A randomized trial comparing three Delphi feedback strategies found no evidence of a difference in a setting with high initial agreement. J Clin Epidemiol. 2018;93:1–8.PubMed
19.
Brookes ST, Chalmers KA, Avery KNL, Coulman K, Blazeby JM, ROMIO study group. Impact of question order on prioritisation of outcomes in the development of a core outcome set: a randomised controlled trial. Trials. 2018;19:66.PubMedPubMedCentral
20.
Gargon E, Crew R, Burnside G, Williamson PR. Higher number of items associated with significantly lower response rates in COS Delphi surveys. J Clin Epidemiol. 2019;108:110–20.PubMedPubMedCentral
21.
De Meyer D, Kottner J, Beele H, Schmitt J, Lange T, Van Hecke A, et al. Delphi procedure in core outcome set development: rating scale and consensus criteria determined outcome selection. J Clin Epidemiol. 2019;111:23–31.PubMed
22.
Lange T, Kopkow C, Lützner J, Günther K-P, Gravius S, Scharf H-P, et al. Comparison of different rating scales for the use in Delphi studies: different scales lead to different consensus and show different test-retest reliability. BMC Med Res Methodol. 2020;20:28.PubMedPubMedCentral
23.
Boel A, Navarro-Compán V, Landewé R, van der Heijde D. Two different invitation approaches for consecutive rounds of a Delphi survey led to comparable final outcome. J Clin Epidemiol. 2021;129:31–9.PubMed
24.
Powell C. The Delphi technique: myths and realities. J Adv Nurs. 2003;41:376–82.PubMed
25.
Schmidt RC. Managing Delphi surveys using nonparametric statistical techniques. Decis Sci. 1997;28:763–74.
26.
Hicks LE. Some properties of ipsative, normative, and forced-choice normative measures. Psychol Bull US: American Psychological Association. 1970;74:167–84.
27.
Feather NT. The measurement of values: Effects of different assessment procedures. Aust J Psychol. 1973;25:221–31 United Kingdom: Taylor & Francis.
28.
Rankin WL, Grube JW. A comparison of ranking and rating procedures for value system measurement. Eur J Soc Psychol. 1980;10:233–46.
29.
Alwin DF, Krosnick JA. The measurement of values in surveys: a comparison of ratings and rankings. Public Opin Q. 1985;49:535–52 United Kingdom: Oxford University Press.
30.
Elwyn G, O’Connor A, Stacey D, Volk R, Edwards A, Coulter A, et al. Developing a quality criteria framework for patient decision aids: online international Delphi consensus process. BMJ. 2006;333:417.PubMedPubMedCentral
31.
Ryan JC, Wiggins B, Edney S, Brinkworth GD, Luscombe-March ND, Carson-Chahhoud KV, et al. Identifying critical features of type two diabetes prevention interventions: A Delphi study with key stakeholders. PLoS ONE. 2021;16:e0255625.PubMedPubMedCentral
32.
de Chiusole D, Stefanutti L. Rating, ranking, or both? A joint application of two probabilistic models for the measurement of values. TPM - Test Psychom Methodol Appl Psychol. 2011;18:49–60.
33.
Moore M. Rating versus ranking in the Rokeach Value Survey: an Israeli comparison. Eur J Soc Psychol. 1975;5:405–8 US: John Wiley & Sons.
34.
van Herk H, van de Velden M. Insight into the relative merits of rating and ranking in a cross-national context using three-way correspondence analysis. Food Qual Prefer. 2007;18:1096–105.
35.
Moors G, Vriens I, Gelissen JPTM, Vermunt JK. Two of a kind. Similarities between ranking and rating data in measuring values. Surv Res Methods. 2016;10:15–33.
36.
Del Grande C, Kaczorowski J, Pomey M-P. What are the top priorities of patients and clinicians for the organization of primary cardiovascular care in Quebec? A modified e-Delphi study. PLoS One. 2023;18:e0280051.PubMedPubMedCentral
37.
Karazivan P, Dumez V, Flora L, Pomey M-P, Del Grande C, Ghadiri DP, et al. The patient-as-partner approach in health care: a conceptual framework for a necessary transition. Acad Med. 2015;90:437–41.PubMed
38.
Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000;32:1008–15.PubMed
39.
Belton I, Wright G, Sissons A, Bolger F, Crawford MM, Hamlin I, et al. Delphi with feedback of rationales: how large can a Delphi group be such that participants are not overloaded, de-motivated, or disengaged? Technol Forecast Soc Change. 2021;170:120897.
40.
World Medical Association. World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects. JAMA. 2013;310:2191–4.
41.
Rowe G, Wright G. Expert opinions in forecasting: the role of the Delphi technique. In: Armstrong JS, editor. Principles of Forecasting: A Handbook for Researchers and Practitioners. Boston, MA: Springer, US; 2001. p. 125–44.
42.
Beckstead JW. On measurements and their quality. Paper 4: verbal anchors and the number of response options in rating scales. Int J Nurs Stud. 2014;51:807–14.PubMed
43.
Toma C. Picioreanu I. The Delphi technique: methodological considerations and the need for reporting guidelines in medical journals. Int J Public Health Res. 2016;4:47–59 Open Science Publishers.
44.
Rowe G, Wright G, Bolger F. Delphi: a reevaluation of research and theory. Technol Forecast Soc Change. 1991;39:235–51.
45.
Krippendorff K. Reliability in content analysis: some common misconceptions and recommendations. Hum Commun Res. 2004;30:411–33.
46.
Hayes AF, Krippendorff K. Answering the call for a standard reliability measure for coding data. Commun Methods Meas. 2007;1:77–89.
47.
48.
Dillman DA, Smyth JD, Christian LM. Internet, phone, mail, and mixed mode surveys: the tailored design method. 4th ed. Hoboken, NJ, US: John Wiley & Sons Inc; 2014.
49.
Krosnick JA. Response strategies for coping with the cognitive demands of attitude measures in surveys. Appl Cogn Psychol. 1991;5:213–36 John Wiley & Sons.
50.
Knäuper B, Belli RF, Hill DH, Herzog AR. Question difficulty and respondents’ cognitive ability: the effect on data quality. JOS. 1997;13:181 Stockholm: Statistics Sweden (SCB).
51.
Richards T, Montori VM, Godlee F, Lapsley P, Paul D. Let the patient revolution begin. BMJ. 2013;346:f2614.PubMed
Metadata
Title
Rating versus ranking in a Delphi survey: a randomized controlled trial
Authors
Claudio Del Grande
Janusz Kaczorowski
Publication date
01-12-2023
Publisher
BioMed Central
Published in
Trials / Issue 1/2023
Electronic ISSN: 1745-6215
DOI
https://doi.org/10.1186/s13063-023-07442-6

Other articles of this Issue 1/2023

Trials 1/2023 Go to the issue

Correction

Correction: Enhancing educational and vocational recovery in adolescents and young adults with early psychosis through Supported Employment and Education (SEEearly): study protocol for a multicenter randomized controlled trial

Study protocol

Comparison of posterior muscle-preserving selective laminectomy and laminectomy with fusion for treating cervical spondylotic myelopathy: study protocol for a randomized controlled trial

Study protocol

Examining the impact of a universal social and emotional learning intervention (Passport) on internalising symptoms and other outcomes among children, compared to the usual school curriculum: study protocol for a school-based cluster randomised trial

Study protocol

The efficacy of a blended intervention to improve physical activity and protein intake for optimal physical recovery after oncological gastrointestinal and lung cancer surgery, the Optimal Physical Recovery After Hospitalization (OPRAH) trial: study protocol for a randomized controlled multicenter trial

Study protocol

Effect of an intraoperative periradicular application of platelet-rich fibrin (PRF) on residual post-surgical neuropathic pain after disc herniation surgery: study protocol for NeuroPRF, a randomized controlled trial

Research

Effect of low-dose aspirin on urinary 11-dehydro-thromboxane B2 in the ASCEND (A Study of Cardiovascular Events iN Diabetes) randomized controlled trial