Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress 2023 EHA Congress 2023 ASCO Annual Meeting
Clinical Collections
Advances in Alzheimer’s

At a glance: The ONWARDS insulin icodec trials

A round-up of the ONWARDS phase 3 clinical trials evaluating the novel weekly insulin icodec in people with diabetes.
ADVANCED SEARCH
Log in
Top
Trials
Published in: Trials 1/2018

Open Access 01-12-2018 | Research

Impact of question order on prioritisation of outcomes in the development of a core outcome set: a randomised controlled trial

Authors: Sara T. Brookes, Katy A. Chalmers, Kerry N. L. Avery, Karen Coulman, Jane M. Blazeby, on behalf of the ROMIO study group

Published in: Trials | Issue 1/2018

Login to get access

Abstract

Background

Core outcome set (COS) developers increasingly employ Delphi surveys to elicit stakeholders’ opinions of which outcomes to measure and report in trials of a particular condition or intervention. Research outside of Delphi surveys and COS development demonstrates that question order can affect response rates and lead to ‘context effects’, where prior questions determine an item’s meaning and influence responses. This study examined the impact of question order within a Delphi survey for a COS for oesophageal cancer surgery.

Methods

A randomised controlled trial was nested within the Delphi survey. Patients and health professionals were randomised to receive a survey including clinical and patient-reported outcomes (PROs), where the PRO section appeared first or last. Participants rated (1–9) the importance of 68 items for inclusion in a COS (ratings 7–9 considered ‘essential’). Analyses considered the impact of question order on: (1) survey response rates; (2) participants’ responses; and (3) items retained at end of the survey.

Results

In total, 116 patients and 71 professionals returned completed surveys. Question order did not affect response rates among patients, but fewer professionals responded when clinical items appeared first (difference = 31.3%, 95% confidence interval [CI] = 13.6–48.9%, P = 0.001). Question order led to different context effects within patients and professionals. While patients rated clinical items highly, irrespective of question order, more PROs were rated essential when appearing last rather than first (difference = 23.7%, 95% CI = 10.5–40.8%). Among professionals, the greatest impact was on clinical items; a higher percentage rated essential when appearing last (difference = 11.6%, 95% CI = 0.0–23.3%). An interaction between question order and the percentage of PRO/clinical items rated essential was observed for patients (P = 0.025) but not professionals (P = 0.357). Items retained for further consideration at the end of the survey were dependent on question order, with discordant items (retained by one question order group only) observed in patients (18/68 [26%]) and professionals (20/68 [29%]).

Conclusions

In the development of a COS, participants’ ratings of potential outcomes within a Delphi survey depend on the context (order) in which the outcomes are asked, consequently impacting on the final COS. Initial piloting is recommended with consideration of the randomisation of items in the survey to reduce potential bias.

Trial registration

The randomised controlled trial reported within this paper was nested within the development of a core outcome set to investigate processes in core outcome set development. Outcomes were not health-related and trial registration was not therefore applicable.
Appendix
Available only for authorised users
Literature
1.
Williamson PR, Gamble C, Altman DG, Hutton JL. Outcome selection bias in meta-analysis. Stat Methods Med Res. 2005;14(5):515–24.CrossRefPubMed
2.
3.
Kirkham JJ, Gargon E, Clarke M, Williamson PR. Can a core outcome set improve the quality of systematic reviews?–a survey of the Co-ordinating Editors of Cochrane Review Groups. Trials. 2013;14(1):21.CrossRefPubMedPubMedCentral
4.
Williamson PR, Altman DG, Blazeby JM, Clarke M, Devane D, Gargon E, et al. Developing core outcome sets for clinical trials: issues to consider. Trials. 2012;13(1):132.CrossRefPubMedPubMedCentral
5.
Sinha IP, Smyth RL, Williamson PR. Using the Delphi technique to determine which outcomes to measure in clinical trials: recommendations for the future based on a systematic review of existing studies. PLoS Med. 2011;8(1):e1000393.CrossRefPubMedPubMedCentral
6.
Gorst SL, Gargon E, Clarke M, Blazeby JM, Altman DG, Williamson PR. Choosing important health outcomes for comparative effectiveness research: an updated review and user survey. PLoS One. 2016;11(1):e0146444.CrossRefPubMedPubMedCentral
7.
8.
Dalkey N, Helmer O. An experimental application of the Delphi method to the use of experts. Manag Sci. 1963;9(3):458–67.CrossRef
9.
McNair AG, Whistance RN, Forsythe RO, Macefield R, Rees J, Pullyblank AM, et al. Core outcomes for colorectal cancer surgery: a consensus study. PLoS Med. 2016;13(8):e1002071.CrossRefPubMedPubMedCentral
10.
Coulman KD, Hopkins J, Brookes ST, Chalmers K, Main B, Owen-Smith A, et al. A core outcome set for the benefits and adverse events of bariatric and metabolic surgery: the BARIACT project. PLoS Med. 2016;13(11):e1002187.CrossRefPubMedPubMedCentral
11.
Harman NL, Bruce IA, Kirkham JJ, Tierney S, Callery P, O’Brien K, et al. The importance of integration of stakeholder views in core outcome set development: otitis media with effusion in children with cleft palate. PLoS One. 2015;10(6):e0129514.CrossRefPubMedPubMedCentral
12.
Brookes ST, Macefield RC, Williamson PR, McNair AG, Potter S, Blencowe NS, et al. Three nested randomized controlled trials of peer-only or multiple stakeholder group feedback within Delphi surveys during core outcome and information set development. Trials. 2016;17(1):409.CrossRefPubMedPubMedCentral
13.
Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000;32(4):1008–15.PubMed
14.
Keeney S, Hasson F, McKenna HP. A critical review of the Delphi technique as a research methodology for nursing. Int J Nurs Stud. 2001;38(2):195–200.CrossRefPubMed
15.
Hsu CC, Sandford BA. The Delphi technique: making sense of consensus. Pract Assess Res Eval. 2007;12(10):1–8.
16.
Hallowell MR, Gambatese JA. Qualitative research: application of the Delphi method to CEM research. J Constr Eng Manag. 2009;136(1):99–107.CrossRef
17.
Landeta J. Current validity of the Delphi method in social sciences. Technol Forecast Soc Change. 2006;73(5):467–82.CrossRef
18.
Sackman H. Delphi Critique. Lexington, MA: Lexington Books; 1975.
19.
Krosnick JA, Presser S. Question and questionnaire design. In: Marsden PV, Wright JD, editors. Handbook of survey research. 2nd ed. Bingley, UK: Emerald; 2010.
20.
McColl E, Jacoby A, Thomas L, Soutter J, Bamford C, Steen N, et al. Design and use of questionnaires: a review of best practice applicable to surveys of health service staff and patients. Health Technol Assess. 2001;5(31):1–256.CrossRefPubMed
21.
Sudman S, Bradburn NM. Asking questions: a practical guide to questionnaire design. San Francisco, CA: Jossey-Bass; 1982.
22.
Jones WH, Lang JR. Sample composition bias and response bias in a mail survey: a comparison of inducement methods. J Market Res. 1980;26(1):69–76.CrossRef
23.
Roberson MT, Sundstrom E. Questionnaire design, return rates, and response favorableness in an employee attitude questionnaire. J Appl Psychol. 1990;75(3):354–7.CrossRef
24.
Nakash RA, Hutton JL, Jørstad-Stein EC, Gates S, Lamb SE. Maximising response to postal questionnaires–a systematic review of randomised trials in health research. BMC Med Res Methodol. 2006;6(1):5.CrossRefPubMedPubMedCentral
25.
Moore DW. Measuring new types of question-order effects: additive and subtractive. Public Opin Q. 2002;66(1):80–91.CrossRef
26.
Schuman H, Presser S. Questions and answers in attitude surveys: experiments on question form, wording, and context. Thousand Oaks, CA: Sage; 1996.
27.
Lasorsa DL. Question-order effects in surveys: the case of political interest, news attention, and knowledge. J Mass Commun Q. 2003;80(3):499–512.
28.
DeMoranville CW, Bienstock CC. Question order effects in measuring service quality. Int J Res Mark. 2003;20(3):217–31.CrossRef
29.
McFarland SG. Effects of question order on survey responses. Public Opin Q. 1981;45(2):208–15.CrossRef
30.
Schuman H, Presser S, Ludwig J. Context effects on survey responses to questions about abortion. Public Opin Q. 1981;45(2):216–23.CrossRef
31.
Tenvergert E, Gillespie MW, Kingma J, Klasen H. Abortion attitudes, 1984-1987-1988: effects of item order and dimensionality. Percept Mot Skills. 1992;74(2):627–42.CrossRefPubMed
32.
33.
Blencowe NS, Strong S, McNair AG, Brookes ST, Crosby T, Griffin SM, et al. Reporting of short-term clinical outcomes after esophagectomy: a systematic review. Ann Surg. 2012;255:658–66.CrossRefPubMed
34.
Macefield RC, Jacobs M, Korfage IJ, Nicklin J, Whistance RN, Brookes ST, et al. Developing core outcome sets: methods for identifying and including patient-reported outcomes (PROs). Trials. 2014;15:49.CrossRefPubMedPubMedCentral
35.
36.
McNair AG, MacKichan F, Donovan JL, Brookes ST, Avery KNL, Griffin SM, et al. What surgeons tell patients and what patients want to know before major cancer surgery: a qualitative study. BMC Cancer. 2016;16:258–65.CrossRefPubMedPubMedCentral
37.
StataCorp. Stata Statistical Software: Release 14. College Station, TX: StataCorp LP; 2015.
38.
Altman DG. Practical statistics for medical research. London: Chapman and Hall; 1991.
39.
Tourangeau R, Rasinski KA, Bradburn N, D’Andrade R. Belief accessibility and context effects in attitude measurement. J Exp Soc Psychol. 1989;25(5):401–21.CrossRef
40.
Bickart BA. Carryover and backfire effects in marketing research. J Market Res. 1993;30(1):52.CrossRef
41.
Coulman KD, Howes N, Hopkins J, Whale K, Chalmers K, Brookes S, et al. A comparison of health professionals’ and patients’ views of the importance of outcomes of bariatric surgery. Obes Surg. 2016;26(11):2738–46.CrossRefPubMedPubMedCentral
42.
Potter S, Brookes ST, Holcombe C, Ward JA, Blazeby JM. Exploring methods for the selection and integration of stakeholder views in the development of core outcome sets: a case study in reconstructive breast surgery. Trials. 2016;17:463.CrossRefPubMedPubMedCentral
43.
Wills GB. Cognitive interviewing: a tool for improving questionnaire design. Thousand Oaks, CA: Sage; 2005.CrossRef
44.
Schwarz N, Schuman H. Political knowledge, attribution, and inferred interest in politics: the operation of buffer items. Int J Public Opinion Res. 1997;9(2):191–5.CrossRef
45.
Perreault WD. Controlling order-effect bias. Public Opin Q. 1975;39(4):544–51.CrossRef
Metadata
Title
Impact of question order on prioritisation of outcomes in the development of a core outcome set: a randomised controlled trial
Authors
Sara T. Brookes
Katy A. Chalmers
Kerry N. L. Avery
Karen Coulman
Jane M. Blazeby
on behalf of the ROMIO study group
Publication date
01-12-2018
Publisher
BioMed Central
Published in
Trials / Issue 1/2018
Electronic ISSN: 1745-6215
DOI
https://doi.org/10.1186/s13063-017-2405-6

Other articles of this Issue 1/2018

Trials 1/2018 Go to the issue

Study protocol

Baduanjin exercise for patients with ischemic heart failure on phase-II cardiac rehabilitation (BEAR trial): study protocol for a prospective randomized controlled trial

Study protocol

Overnight auto-adjusting continuous airway pressure + standard care compared with standard care alone in the prevention of morbidity in sickle cell disease phase II (POMS2b): study protocol for a randomised controlled trial

Study protocol

Administration of taurolidine-citrate lock solution for prevention of central venous catheter infection in adult neutropenic haematological patients: a randomised, double-blinded, placebo-controlled trial (TAURCAT)

Study protocol

Intravenous morphine versus intravenous paracetamol after cardiac surgery in neonates and infants: a study protocol for a randomized controlled trial

Study protocol

The effects of an activity-based lifestyle intervention on moderate sleep complaints among older adults: study protocol for a randomized controlled trial

Study protocol

Prophylactic platelet transfusion prior to central venous catheter placement in patients with thrombocytopenia: study protocol for a randomised controlled trial