Skip to main content
Top
Published in: Trials 1/2010

Open Access 01-12-2010 | Commentary

Obstacles to researching the researchers: A case study of the ethical challenges of undertaking methodological research investigating the reporting of randomised controlled trials

Authors: Joanne E McKenzie, G Peter Herbison, Paul Roth, Charlotte Paul

Published in: Trials | Issue 1/2010

Login to get access

Abstract

Background

Recent cohort studies of randomised controlled trials have provided evidence of within-study selective reporting bias; where statistically significant outcomes are more likely to be more completely reported compared to non-significant outcomes. Bias resulting from selective reporting can impact on meta-analyses, influencing the conclusions of systematic reviews, and in turn, evidence based clinical practice guidelines.
In 2006 we received funding to investigate if there was evidence of within-study selective reporting in a cohort of RCTs submitted to New Zealand Regional Ethics Committees in 1998/99. This research involved accessing ethics applications, their amendments and annual reports, and comparing these with corresponding publications. We did not plan to obtain informed consent from trialists to view their ethics applications for practical and scientific reasons.
In November 2006 we sought ethical approval to undertake the research from our institutional ethics committee. The Committee declined our application on the grounds that we were not obtaining informed consent from the trialists to view their ethics application. This initiated a seventeen month process to obtain ethical approval. This publication outlines what we planned to do, the issues we encountered, discusses the legal and ethical issues, and presents some potential solutions.

Discussion and conclusion

Methodological research such as this has the potential for public benefit and there is little or no harm for the participants (trialists) in undertaking it. Further, in New Zealand, there is freedom of information legislation, which in this circumstance, unambiguously provided rights of access and use of the information in the ethics applications. The decision of our institutional ethics committee defeated this right and did not recognise the nature of this observational research.
Methodological research, such as this, can be used to develop processes to improve quality in research reporting. Recognition of the potential benefit of this research in the broader research community, and those who sit on ethics committees, is perhaps needed. In addition, changes to the ethical review process which involve separation between those who review proposals to undertake methodological research using ethics applications, and those with responsibility for reviewing ethics applications for trials, should be considered. Finally, we contend that the research community could benefit from quality improvement approaches used in allied sectors.
Appendix
Available only for authorised users
Literature
1.
go back to reference Grimes DA, Schulz KF: An overview of clinical research: the lay of the land. Lancet. 2002, 359: 57-61. 10.1016/S0140-6736(02)07283-5.CrossRefPubMed Grimes DA, Schulz KF: An overview of clinical research: the lay of the land. Lancet. 2002, 359: 57-61. 10.1016/S0140-6736(02)07283-5.CrossRefPubMed
2.
go back to reference Agha R, Cooper D, Muir G: The reporting quality of randomised controlled trials in surgery: a systematic review. Int J Surg. 2007, 5: 413-422. 10.1016/j.ijsu.2007.06.002.CrossRefPubMed Agha R, Cooper D, Muir G: The reporting quality of randomised controlled trials in surgery: a systematic review. Int J Surg. 2007, 5: 413-422. 10.1016/j.ijsu.2007.06.002.CrossRefPubMed
3.
go back to reference Cowan J, Lozano-Calderon S, Ring D: Quality of prospective controlled randomized trials. Analysis of trials of treatment for lateral epicondylitis as an example. J Bone Joint Surg Am. 2007, 89: 1693-1699. 10.2106/JBJS.F.00858.CrossRefPubMed Cowan J, Lozano-Calderon S, Ring D: Quality of prospective controlled randomized trials. Analysis of trials of treatment for lateral epicondylitis as an example. J Bone Joint Surg Am. 2007, 89: 1693-1699. 10.2106/JBJS.F.00858.CrossRefPubMed
4.
go back to reference Farrokhyar F, Chu R, Whitlock R, Thabane L: A systematic review of the quality of publications reporting coronary artery bypass grafting trials. Can J Surg. 2007, 50: 266-277.PubMedPubMedCentral Farrokhyar F, Chu R, Whitlock R, Thabane L: A systematic review of the quality of publications reporting coronary artery bypass grafting trials. Can J Surg. 2007, 50: 266-277.PubMedPubMedCentral
5.
go back to reference Kane RL, Wang J, Garrard J: Reporting in randomized clinical trials improved after adoption of the CONSORT statement. J Clin Epidemiol. 2007, 60: 241-249. 10.1016/j.jclinepi.2006.06.016.CrossRefPubMed Kane RL, Wang J, Garrard J: Reporting in randomized clinical trials improved after adoption of the CONSORT statement. J Clin Epidemiol. 2007, 60: 241-249. 10.1016/j.jclinepi.2006.06.016.CrossRefPubMed
6.
go back to reference Thabane L, Chu R, Cuddy K, Douketis J: What is the quality of reporting in weight loss intervention studies? A systematic review of randomized controlled trials. Int J Obes (Lond). 2007, 31: 1554-1559. 10.1038/sj.ijo.0803640.CrossRef Thabane L, Chu R, Cuddy K, Douketis J: What is the quality of reporting in weight loss intervention studies? A systematic review of randomized controlled trials. Int J Obes (Lond). 2007, 31: 1554-1559. 10.1038/sj.ijo.0803640.CrossRef
7.
go back to reference Juni P, Altman DG, Egger M: Systematic reviews in health care: Assessing the quality of controlled clinical trials. BMJ. 2001, 323: 42-46. 10.1136/bmj.323.7303.42.CrossRefPubMedPubMedCentral Juni P, Altman DG, Egger M: Systematic reviews in health care: Assessing the quality of controlled clinical trials. BMJ. 2001, 323: 42-46. 10.1136/bmj.323.7303.42.CrossRefPubMedPubMedCentral
8.
go back to reference Schulz KF, Chalmers I, Hayes RJ, Altman DG: Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995, 273: 408-412. 10.1001/jama.273.5.408.CrossRefPubMed Schulz KF, Chalmers I, Hayes RJ, Altman DG: Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995, 273: 408-412. 10.1001/jama.273.5.408.CrossRefPubMed
9.
go back to reference Pildal J, Hrobjartsson A, Jorgensen KJ, Hilden J, Altman DG, Gotzsche PC: Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol. 2007, 36: 847-857. 10.1093/ije/dym087.CrossRefPubMed Pildal J, Hrobjartsson A, Jorgensen KJ, Hilden J, Altman DG, Gotzsche PC: Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol. 2007, 36: 847-857. 10.1093/ije/dym087.CrossRefPubMed
10.
go back to reference Wood L, Egger M, Gluud LL, Schulz KF, Juni P, Altman DG, Gluud C, Martin RM, Wood AJ, Sterne JA: Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ. 2008, 336: 601-605. 10.1136/bmj.39465.451748.AD.CrossRefPubMedPubMedCentral Wood L, Egger M, Gluud LL, Schulz KF, Juni P, Altman DG, Gluud C, Martin RM, Wood AJ, Sterne JA: Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ. 2008, 336: 601-605. 10.1136/bmj.39465.451748.AD.CrossRefPubMedPubMedCentral
11.
go back to reference Chan AW, Altman DG: Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors. BMJ. 2005, 330: 753-10.1136/bmj.38356.424606.8F.CrossRefPubMedPubMedCentral Chan AW, Altman DG: Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors. BMJ. 2005, 330: 753-10.1136/bmj.38356.424606.8F.CrossRefPubMedPubMedCentral
12.
go back to reference Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG: Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004, 291: 2457-2465. 10.1001/jama.291.20.2457.CrossRefPubMed Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG: Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004, 291: 2457-2465. 10.1001/jama.291.20.2457.CrossRefPubMed
13.
go back to reference Chan AW, Krleza-Jeric K, Schmid I, Altman DG: Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ. 2004, 171: 735-740.CrossRefPubMedPubMedCentral Chan AW, Krleza-Jeric K, Schmid I, Altman DG: Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ. 2004, 171: 735-740.CrossRefPubMedPubMedCentral
14.
go back to reference Furukawa TA, Watanabe N, Omori IM, Montori VM, Guyatt GH: Association between unreported outcomes and effect size estimates in Cochrane meta-analyses. JAMA. 2007, 297: 468-470. 10.1001/jama.297.5.468-b.CrossRefPubMed Furukawa TA, Watanabe N, Omori IM, Montori VM, Guyatt GH: Association between unreported outcomes and effect size estimates in Cochrane meta-analyses. JAMA. 2007, 297: 468-470. 10.1001/jama.297.5.468-b.CrossRefPubMed
15.
go back to reference Al-Marzouki S, Roberts I, Evans S, Marshall T: Selective reporting in clinical trials: analysis of trial protocols accepted by The Lancet. Lancet. 2008, 372: 201-10.1016/S0140-6736(08)61060-0.CrossRefPubMed Al-Marzouki S, Roberts I, Evans S, Marshall T: Selective reporting in clinical trials: analysis of trial protocols accepted by The Lancet. Lancet. 2008, 372: 201-10.1016/S0140-6736(08)61060-0.CrossRefPubMed
16.
go back to reference Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R: Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med. 2008, 358: 252-260. 10.1056/NEJMsa065779.CrossRefPubMed Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R: Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med. 2008, 358: 252-260. 10.1056/NEJMsa065779.CrossRefPubMed
17.
go back to reference Rising K, Bacchetti P, Bero L: Reporting Bias in Drug Trials Submitted to the Food and Drug Administration: Review of Publication and Presentation. PLoS Med. 2008, 5: e217-10.1371/journal.pmed.0050217.CrossRefPubMedPubMedCentral Rising K, Bacchetti P, Bero L: Reporting Bias in Drug Trials Submitted to the Food and Drug Administration: Review of Publication and Presentation. PLoS Med. 2008, 5: e217-10.1371/journal.pmed.0050217.CrossRefPubMedPubMedCentral
18.
go back to reference Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan AW, Cronin E, Decullier E, Easterbrook PJ, Von Elm E, Gamble C, Ghersi D, Ioannidis JP, Simes J, Williamson PR: Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE. 2008, 3: e3081-10.1371/journal.pone.0003081.CrossRefPubMedPubMedCentral Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan AW, Cronin E, Decullier E, Easterbrook PJ, Von Elm E, Gamble C, Ghersi D, Ioannidis JP, Simes J, Williamson PR: Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE. 2008, 3: e3081-10.1371/journal.pone.0003081.CrossRefPubMedPubMedCentral
19.
go back to reference Williamson PR: Identification and impact of outcome selection bias in meta-analysis. Stat Med. 2005, 24: 1547-1561. 10.1002/sim.2025.CrossRefPubMed Williamson PR: Identification and impact of outcome selection bias in meta-analysis. Stat Med. 2005, 24: 1547-1561. 10.1002/sim.2025.CrossRefPubMed
20.
go back to reference Williamson PR, Gamble C, Altman DG, Hutton JL: Outcome selection bias in meta-analysis. Stat Methods Med Res. 2005, 14: 515-524. 10.1191/0962280205sm415oa.CrossRefPubMed Williamson PR, Gamble C, Altman DG, Hutton JL: Outcome selection bias in meta-analysis. Stat Methods Med Res. 2005, 14: 515-524. 10.1191/0962280205sm415oa.CrossRefPubMed
21.
go back to reference Moher D: Reporting research results: a moral obligation for all researchers. Can J Anaesth. 2007, 54: 331-335. 10.1007/BF03022653.CrossRefPubMed Moher D: Reporting research results: a moral obligation for all researchers. Can J Anaesth. 2007, 54: 331-335. 10.1007/BF03022653.CrossRefPubMed
22.
go back to reference Ramsey S, Scoggins J: Commentary: practicing on the tip of an information iceberg? Evidence of underpublication of registered clinical trials in oncology. Oncologist. 2008, 13: 925-929. 10.1634/theoncologist.2008-0133.CrossRefPubMedPubMedCentral Ramsey S, Scoggins J: Commentary: practicing on the tip of an information iceberg? Evidence of underpublication of registered clinical trials in oncology. Oncologist. 2008, 13: 925-929. 10.1634/theoncologist.2008-0133.CrossRefPubMedPubMedCentral
23.
24.
go back to reference Altman D, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne DR, Gotzsche PC, Lang T: The revised CONSORT Statement for Reporting Randomized Traials: Explanation and Elaboration. Ann Intern Med. 2001, 134: 663-694.CrossRefPubMed Altman D, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne DR, Gotzsche PC, Lang T: The revised CONSORT Statement for Reporting Randomized Traials: Explanation and Elaboration. Ann Intern Med. 2001, 134: 663-694.CrossRefPubMed
25.
go back to reference Hahn S, Williamson PR, Hutton JL: Investigation of within-study selective reporting in clinical research: follow-up of applications submitted to a local research ethics committee. J Eval Clin Pract. 2002, 8: 353-359. 10.1046/j.1365-2753.2002.00314.x.CrossRefPubMed Hahn S, Williamson PR, Hutton JL: Investigation of within-study selective reporting in clinical research: follow-up of applications submitted to a local research ethics committee. J Eval Clin Pract. 2002, 8: 353-359. 10.1046/j.1365-2753.2002.00314.x.CrossRefPubMed
27.
go back to reference Health Research Council: Guidelines on Ethics in Health Research. 2002, Health Research Council, (Revised 2005) Health Research Council: Guidelines on Ethics in Health Research. 2002, Health Research Council, (Revised 2005)
28.
go back to reference National Ethics Advisory Committee: Ethical guidelines for observational studies: Observational research, audits and related activities. 2006, Wellington: Ministry of Health National Ethics Advisory Committee: Ethical guidelines for observational studies: Observational research, audits and related activities. 2006, Wellington: Ministry of Health
29.
go back to reference Evans S: When and how can endpoints be changed after initiation of a randomized clinical trial?. PLoS Clin Trials. 2007, 2: e18-10.1371/journal.pctr.0020018.CrossRefPubMedPubMedCentral Evans S: When and how can endpoints be changed after initiation of a randomized clinical trial?. PLoS Clin Trials. 2007, 2: e18-10.1371/journal.pctr.0020018.CrossRefPubMedPubMedCentral
30.
go back to reference Sim I, Chan AW, Gulmezoglu AM, Evans T, Pang T: Clinical trial registration: transparency is the watchword. Lancet. 2006, 367: 1631-1633. 10.1016/S0140-6736(06)68708-4.CrossRefPubMed Sim I, Chan AW, Gulmezoglu AM, Evans T, Pang T: Clinical trial registration: transparency is the watchword. Lancet. 2006, 367: 1631-1633. 10.1016/S0140-6736(06)68708-4.CrossRefPubMed
31.
go back to reference Wendler D, Krohmal B, Emanuel EJ, Grady C: Why patients continue to participate in clinical research. Arch Intern Med. 2008, 168: 1294-1299. 10.1001/archinte.168.12.1294.CrossRefPubMed Wendler D, Krohmal B, Emanuel EJ, Grady C: Why patients continue to participate in clinical research. Arch Intern Med. 2008, 168: 1294-1299. 10.1001/archinte.168.12.1294.CrossRefPubMed
32.
go back to reference Chan AW, Upshur R, Singh JA, Ghersi D, Chapuis F, Altman DG: Research protocols: Waiving confidentiality for the greater good. BMJ. 2006, 332: 1086-1089. 10.1136/bmj.332.7549.1086.CrossRefPubMedPubMedCentral Chan AW, Upshur R, Singh JA, Ghersi D, Chapuis F, Altman DG: Research protocols: Waiving confidentiality for the greater good. BMJ. 2006, 332: 1086-1089. 10.1136/bmj.332.7549.1086.CrossRefPubMedPubMedCentral
33.
go back to reference De Angelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, Kotzin S, Laine C, Marusic A, Overbeke AJ, Schroeder TV, Sox HC, Weyden Van Der MB: Clinical trial registration: a statement from the International Committee of Medical Journal Editors. Ann Intern Med. 2004, 141: 477-478.CrossRefPubMed De Angelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, Kotzin S, Laine C, Marusic A, Overbeke AJ, Schroeder TV, Sox HC, Weyden Van Der MB: Clinical trial registration: a statement from the International Committee of Medical Journal Editors. Ann Intern Med. 2004, 141: 477-478.CrossRefPubMed
36.
go back to reference United States Code (2008) US Public Law 110-85: Food and Drug Administration Amendments Act of 2007. United States Code (2008) US Public Law 110-85: Food and Drug Administration Amendments Act of 2007.
37.
go back to reference Plint AC, Moher D, Morrison A, Schulz K, Altman DG, Hill C, Gaboury I: Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. Med J Aust. 2006, 185: 263-267.PubMed Plint AC, Moher D, Morrison A, Schulz K, Altman DG, Hill C, Gaboury I: Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. Med J Aust. 2006, 185: 263-267.PubMed
38.
go back to reference Jonsen AR: Of balloons and bicycles or the relationship between ethical theory and practical judgment. Hastings Cent Rep. 1991, 21: 14-17. 10.2307/3562885.CrossRefPubMed Jonsen AR: Of balloons and bicycles or the relationship between ethical theory and practical judgment. Hastings Cent Rep. 1991, 21: 14-17. 10.2307/3562885.CrossRefPubMed
39.
go back to reference Paul C: Health researchers' views of ethics committee functioning in New Zealand. N Z Med J. 2000, 113: 210-214.PubMed Paul C: Health researchers' views of ethics committee functioning in New Zealand. N Z Med J. 2000, 113: 210-214.PubMed
40.
go back to reference Hughes FJ: Assessing the effects of bias on the magnitude of outcomes in clinical studies: lessons for the research community?. J Clin Periodontol. 2008, 35: 773-774. 10.1111/j.1600-051X.2008.01300.x.CrossRefPubMed Hughes FJ: Assessing the effects of bias on the magnitude of outcomes in clinical studies: lessons for the research community?. J Clin Periodontol. 2008, 35: 773-774. 10.1111/j.1600-051X.2008.01300.x.CrossRefPubMed
41.
go back to reference Angell EL, Bryman A, Ashcroft RE, Dixon-Woods M: An analysis of decision letters by research ethics committees: the ethics/scientific quality boundary examined. Qual Saf Health Care. 2008, 17: 131-136. 10.1136/qshc.2007.022756.CrossRefPubMed Angell EL, Bryman A, Ashcroft RE, Dixon-Woods M: An analysis of decision letters by research ethics committees: the ethics/scientific quality boundary examined. Qual Saf Health Care. 2008, 17: 131-136. 10.1136/qshc.2007.022756.CrossRefPubMed
42.
go back to reference Babl FE, Sharwood LN: Research governance: current knowledge among clinical researchers. Med J Aust. 2008, 188: 649-652.PubMed Babl FE, Sharwood LN: Research governance: current knowledge among clinical researchers. Med J Aust. 2008, 188: 649-652.PubMed
43.
44.
go back to reference Schüklenk U, Lott JP: Bioethics and (Public) Policy Advice. Bioethics in a Small World. Edited by: Thiele F, Ashcroft RE, Wèutscher F. 2005, Berlin, Heidelberg: Springer-Verlag Berlin Heidelberg Schüklenk U, Lott JP: Bioethics and (Public) Policy Advice. Bioethics in a Small World. Edited by: Thiele F, Ashcroft RE, Wèutscher F. 2005, Berlin, Heidelberg: Springer-Verlag Berlin Heidelberg
45.
Metadata
Title
Obstacles to researching the researchers: A case study of the ethical challenges of undertaking methodological research investigating the reporting of randomised controlled trials
Authors
Joanne E McKenzie
G Peter Herbison
Paul Roth
Charlotte Paul
Publication date
01-12-2010
Publisher
BioMed Central
Published in
Trials / Issue 1/2010
Electronic ISSN: 1745-6215
DOI
https://doi.org/10.1186/1745-6215-11-28

Other articles of this Issue 1/2010

Trials 1/2010 Go to the issue