Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.
ADVANCED SEARCH
Log in
Top
BMC Medical Research Methodology
Published in: BMC Medical Research Methodology 1/2005

Open Access 01-12-2005 | Research article

Stability of response characteristics of a Delphi panel: application of bootstrap data expansion

Authors: Ralitsa B Akins, Homer Tolson, Bryan R Cole

Published in: BMC Medical Research Methodology | Issue 1/2005

Login to get access

Abstract

Background

Delphi surveys with panels of experts in a particular area of interest have been widely utilized in the fields of clinical medicine, nursing practice, medical education and healthcare services. Despite this wide applicability of the Delphi methodology, there is no clear identification of what constitutes a sufficient number of Delphi survey participants to ensure stability of results.

Methods

The study analyzed the response characteristics from the first round of a Delphi survey conducted with 23 experts in healthcare quality and patient safety. The panel members had similar training and subject matter understanding of the Malcolm Baldrige Criteria for Performance Excellence in Healthcare. The raw data from the first round sampling, which usually contains the largest diversity of responses, were augmented via bootstrap sampling to obtain computer-generated results for two larger samples obtained by sampling with replacement. Response characteristics (mean, trimmed mean, standard deviation and 95% confidence intervals) for 54 survey items were compared for the responses of the 23 actual study participants and two computer-generated samples of 1000 and 2000 resampling iterations.

Results

The results from this study indicate that the response characteristics of a small expert panel in a well-defined knowledge area are stable in light of augmented sampling.

Conclusion

Panels of similarly trained experts (who possess a general understanding in the field of interest) provide effective and reliable utilization of a small sample from a limited number of experts in a field of study to develop reliable criteria that inform judgment and support effective decision-making.
Literature
1.
Williams PL, Webb C: The Delphi technique: a methodological discussion. J Adv Nurs. 1994, 19: 180-6.CrossRefPubMed
2.
Fischer RG: The Delphi method: a description, review and criticism. J Acad Librariansh. 1978, 4 (2): 64-70.
3.
Wilhelm WJ: Alchemy of the Oracle: the Delphi technique. The Delta Pi Epsilon Journal. 2001, 43 (1): 6-26.
4.
Reid NG: The Delphi technique: its contribution to the evaluation of professional practice. Professional competence and quality assurance in the caring professions. Edited by: Ellis R. 1988, New York, NY: Chapman and Hall
5.
Malone DC, Abarca J, Hansten PD, Grizzle AJ, Armstrong EP, Van Bergen RC, Duncan-Edgar BS, Solomon SL, Lipton RB: Identification of serious drug-drug interactions: results of the partnership to prevent drug-drug interactions. Am J Geriatr Pharmacother. 2005, 3 (2): 65-76. 10.1016/j.amjopharm.2005.05.001.CrossRef
6.
Strasser S, London L, Kortenbout E: Developing a competence framework and evaluation tool for primary care nurses in South Africa. Educ Health. 2005, 18 (2): 133-144.CrossRef
7.
De Vet E, Brug J, De Nooijer J, Dijkstra A, De Vries NK: Determinants of forward stage transitions: a Delphi study. Health Educ Res. 2005, 20 (2): 195-205. 10.1093/her/cyg111.CrossRefPubMed
8.
Howell S, Kemp C: Defining early number sense: a participatory Australian study. J Educ Psychol. 2005, 25 (5): 555-571. 10.1080/01443410500046838.CrossRef
9.
Nekolaichuk CL, Fainsinger RL, Lawlor PG: A validation study of a pain classification system for advanced cancer patients using content experts: the Edmonton Classification System for cancer pain. Palliat Med. 2005, 19: 466-476. 10.1191/0269216305pm1055oa.CrossRefPubMed
10.
Armstrong D, Marshall JK, Chiba N, Enns R, Fallone CA, Fass R, Hollingworth R, Hunt RH, Kahrilas PJ, Mayrand S, Moayyedi P, Paterson WG, Sadowski D, van Zanten SJ: Canadian consensus conference on the management of gastroesophageal reflux disease in adults – update 2004. Can J Gastroenterol. 2005, 19 (1): 15-35.CrossRefPubMed
11.
Stolee P, Esbaugh J, Aylward S, Cathers T, Harvey DP, Hillier LM, Keat N, Feightner JW: Factors associated with the effectiveness of continuing education in long-term care. Gerontologist. 2005, 45 (3): 399-405.CrossRefPubMed
12.
Al-Marzouki S, Roberts I, Marshall T, Evans S: The effect of scientific misconduct on the results of clinical trials: a Delphi survey. Contemp Clin Trials. 2005, 26 (3): 331-337. 10.1016/j.cct.2005.01.011.CrossRefPubMed
13.
McKenna H, Keeney S, Bradley M: Nurse leadership within primary care: The perceptions of community nurses, GPs, policy makers and members of the public. J Nurs Manag. 2004, 12: 69-76.CrossRefPubMed
14.
Mcilfatrick SJ, Keeney S: Identifying cancer nursing research priorities using the Delphi technique. J Adv Nurs. 2003, 42 (6): 629-636. 10.1046/j.1365-2648.2003.02666.x.CrossRefPubMed
15.
Coico R, Kachur E, Lima V, Lipper S: Guidelines for preklership bioterrorism curricula. Acad Med. 2004, 79 (4): 366-375. 10.1097/00001888-200404000-00019.CrossRefPubMed
16.
Telford R, Boote JD, Cooper CL: What does it mean to involve consumers successfully in NHS research? A consensus study. Health Expect. 2004, 7 (3): 209-220. 10.1111/j.1369-7625.2004.00278.x.CrossRefPubMed
17.
Meadows AB, Maine LL, Keyes EK, Pearson K, Finstuen K: Pharmacy executive leadership issues and associated skills, knowledge and abilities. J Am Pharm Assoc. 2005, 45 (1): 55-62. 10.1331/1544345052843183.CrossRef
18.
Kelly KP, Porock D: A survey of paediatric oncology nurses' perceptions of parent educational needs. J Pediatr Oncol Nurs. 2005, 22 (1): 58-66. 10.1177/1043454204272537.CrossRefPubMed
19.
Cypert FR, Gant WL: The Delphi technique: A case study. Phi Delta Kappan. 1971, 272-273.
20.
Barnette JJ, Danielson LC, Algozzine RF: Delphi methodology: an empirical investigation. Educational Research Quarterly. 1978, 3 (1): 67-73.
21.
Synnott G, McKie D: International issues in PR: Researching research and prioritising priorities. Journal of Public Relations Research. 1997, 9 (4): 259-282. 10.1207/s1532754xjprr0904_02.CrossRef
22.
Hasson F, Keeney S, McKenna H: Research guidelines for the Delphi survey technique. J Adv Nurs. 2000, 32 (4): 1008-1015. 10.1046/j.1365-2648.2000.01567.x.PubMed
23.
Robinson AR, Hohmann KB, Rifkin JI, Topp D, Gilroy CM, Pickard JR, Anderson RJ: Physician and public opinions on quality of health care and the problem of medical errors. Arch Intern Med. 2002, 162 (19): 2186-2190. 10.1001/archinte.162.19.2186.CrossRefPubMed
24.
Sexton JB, Helmreih RL, Rowan K, Vella K, Boyden J, Neilands TB, Roberts PR, Thomas EJ: The safety attitude questionnaire: a psychometric validation. Technical report. 2003, The University of Texas Center of Excellence for Patient Safety Research and Practice, Austin, Texas, AHRQ Grants 1PO1HS1154401 and U18HS1116401
25.
26.
Linstone HA, Turoff M: The Delphi Method: techniques and applications. 1975, Reading, MA: Addison-Wesley Publishing Company
27.
Adler M, Ziglio E: Gazing into the Oracle: the Delphi method and its application to social policy and public health. 1996, Bristol, PA: Jessica Kingsley Publishers, Ltd
28.
Rotondi A, Gustafson D: Theoretical, methodological and practical issues arising out of the Delphi method. Gazing into the Oracle: the Delphi method and its application to social policy and public health. Edited by: Adler M, Ziglio E. 1996, Bristol, PA: Jessica Kingsley Publishers, Ltd, 34-55.
29.
30.
Efron B, Tibshirani R: An introduction to the bootstrap. Monographs on statistics and applied probability, No 57. 1993, London: Chapman and Hall
31.
Edgington ES: Randomization tests. 1995, New York: M. Dekker
32.
Good P: Permutation tests: a practical guide for resampling methods for testing hypotheses. 2000, New York: Springer-VerlagCrossRef
33.
Henderson AR: The bootstrap: a technique for data-driven statistics. Using computer intensive analyses to explore experimental data. Clin Chim Acta. 2005, 359: 1-26. 10.1016/j.cccn.2005.04.002.CrossRefPubMed
34.
Linden A, Adams JL, Roberts N: Evaluating disease management program effectiveness: an introduction to the bootstrap technique. Disease Management & Health Outcomes. 2005, 13 (3): 159-167.CrossRef
35.
Lloyd CJ: On comparing the accuracy of competing tests of the same hypotheses of simulation data. J Stat Plan Inference. 2005, 128: 497-508. 10.1016/j.jspi.2003.12.002.CrossRef
36.
Calzone KA, Jenkins J, Masny A: Core competencies in cancer genetics for advanced practice oncology nurses. Oncol Nurs Forum. 2002, 29 (9): 1327-1333.CrossRefPubMed
37.
Wang CC, Wang Y, Zhang K, Fang J, Liu W, Luo S, Tang S, Wang S, Li VC: Reproductive health indicators for China rural areas. Soc Sci Med. 2003, 57: 217-225. 10.1016/S0277-9536(02)00341-6.CrossRefPubMed
38.
Catlin AJ, Carter BS: Creation of a neonatal end-of-life palliative care protocol. J Perinat Neonal Nurs. 2002, 21 (4): 37-49.
39.
Sharp D, Liebenau A, Stocks N, Bennewith O, Evans M, Jones WB, Peters T, Goldberg D, Gunnell D: Locally developed guidelines for the aftercare of deliberate self-harm patients in general practice. Primary Health Care Research and Development. 2003, 4: 21-28. 10.1191/1463423603pc119oa.CrossRef
40.
Escobar A, Qintana JM, Arostegui I, Azkarate J, Guenaga JI, Arenaza JC, Garai I: Development of explicit criteria for total knee replacement. Int J Technol Assess Health Care. 2003, 19 (1): 57-70. 10.1017/S0266462303000060.CrossRefPubMed
41.
McBride AJ, Pates R, Ramadan R, McGowan C: Delphi survey of experts' opinions on strategies used by community pharmacists to reduce over-the-counter drug misuse. Addiction. 2003, 98: 487-497. 10.1046/j.1360-0443.2003.00345.x.CrossRefPubMed
42.
Jones J, Hunter D: Consensus methods for medical and health services research. Br Med J. 1995, 311 (7001): 376-380.CrossRef
43.
Elnicki DM, Lescisin DA, Case S: Improving the National Board of Medical Examiners Internal Medicine Subject Exam for use in clerkship evaluation. J Gen Intern Med. 2002, 17: 435-440. 10.1046/j.1525-1497.2002.10673.x.CrossRefPubMedPubMedCentral
44.
Ziglio E: The Delphi method and its contribution to decision making. Gazing into the Oracle: the Delphi method and its application to social policy and public health. Edited by: Adler M, Ziglio E. 1996, Bristol, PA: Jessica Kingsley Publishers, Ltd, 3-33.
45.
Bernstein SJ, Laouri M, Hilborne LH, Leape LL, Kahan JP, Park RE, Kamberg CJ, Brook RH: Coronary Angiography: a literature review and ratings of appropriateness and necessity. 1992, The New York City Rand Institute, JRA-03. Santa Monica, CA: The Rand Corporation
46.
Maetzel A, Strand V, Tugwell P, Wells G, Bombardier C: Economic comparison of Leflunomide and Methotrexate in patients with rheumatoid arthritis. Pharmacoeconomics. 2002, 20 (1): 61-70.CrossRefPubMed
47.
Manktelow B, Abrams K, Draper E: A comparison of six methods of estimating confidence intervals for an indirectly standardized outcome ratio. J Epidemiol Community Health. A36-Suppl 1
48.
Peris F, Martinez E, Badia X, Brosa M: Iatrogenic cost factors incorporating mild and moderate adverse events in the economic comparison of Aceclofenac and other NSAIDs. Pharmacoeconomics. 2001, 19 (7): 779-790.CrossRefPubMed
49.
Neymark N, Adriaenssen I, Gorlia T, Caleo S, Bolla M, Brochon D: Cost-effectiveness of the addition of early hormonal therapy in locally advanced prostate cancer: results decisively determined by the cut-off time point chosen for the analysis. Eur J Cancer. 2001, 37: 1768-1774. 10.1016/S0959-8049(01)00197-6.CrossRefPubMed
50.
Chen M, Kianifard F: A nonparametric procedure associated with a clinically meaningful efficacy measure. Biostatistics. 2000, 1: 293-298. 10.1093/biostatistics/1.3.293.CrossRefPubMed
51.
Bredell H, Crookes RL, Heynes du PA, Schoub BD, Morris L: Molecular investigation of two possible cases of accidental HIV-1 transmission in South Africa. AIDS Res Hum Retroviruses. 2003, 19 (7): 613-617. 10.1089/088922203322230987.CrossRefPubMed
52.
MacNab YC, Farrel PJ, Gustafson P, Wen S: Estimation in Bayesian disease mapping. Biometrics. 2004, 60: 865-873. 10.1111/j.0006-341X.2004.00241.x.CrossRefPubMed
Metadata
Title
Stability of response characteristics of a Delphi panel: application of bootstrap data expansion
Authors
Ralitsa B Akins
Homer Tolson
Bryan R Cole
Publication date
01-12-2005
Publisher
BioMed Central
Published in
BMC Medical Research Methodology / Issue 1/2005
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/1471-2288-5-37

Other articles of this Issue 1/2005

BMC Medical Research Methodology 1/2005 Go to the issue

Research article

The strengths and limitations of meta-analyses based on aggregate data

Debate

Systems validation: application to statistical programs

Debate

Assessing subgroup effects with binary data: can the use of different effect measures lead to different conclusions?

Research article

Use of re-randomized data in meta-analysis

Research article

No role for quality scores in systematic reviews of diagnostic accuracy studies

Research article

Modeling repeated ordinal responses using a family of power transformations: application to neonatal hypothermia data