Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress 2023 EHA Congress 2023 ASCO Annual Meeting
Clinical Collections
Advances in Alzheimer’s

At a glance: The ONWARDS insulin icodec trials

A round-up of the ONWARDS phase 3 clinical trials evaluating the novel weekly insulin icodec in people with diabetes.
ADVANCED SEARCH
Log in
Top
Systematic Reviews
Published in: Systematic Reviews 1/2018

Open Access 01-12-2018 | Methodology

Applying the ROBINS-I tool to natural experiments: an example from public health

Authors: Hilary Thomson, Peter Craig, Michele Hilton-Boon, Mhairi Campbell, Srinivasa Vittal Katikireddi

Published in: Systematic Reviews | Issue 1/2018

Login to get access

Abstract

Background

A new tool to assess Risk of Bias In Non-randomised Studies of Interventions (ROBINS-I) was published in Autumn 2016. ROBINS-I uses the Cochrane-approved risk of bias (RoB) approach and focusses on internal validity. As such, ROBINS-I represents an important development for those conducting systematic reviews which include non-randomised studies (NRS), including public health researchers. We aimed to establish the applicability of ROBINS-I using a group of NRS which have evaluated non-clinical public health natural experiments.

Methods

Five researchers, all experienced in critical appraisal of non-randomised studies, used ROBINS-I to independently assess risk of bias in five studies which had assessed the health impacts of a domestic energy efficiency intervention. ROBINS-I assessments for each study were entered into a database and checked for consensus across the group. Group discussions were used to identify reasons underpinning lack of consensus for specific questions and bias domains.

Results

ROBINS-I helped to systematically articulate sources of bias in NRS. However, the lack of consensus in assessments for all seven bias domains raised questions about ROBINS-I’s reliability and applicability for natural experiment studies. The two RoB domains with least consensus were selection (Domain 2) and performance (Domain 4). Underlying the lack of consensus were difficulties in applying an intention to treat or per protocol effect of interest to the studies. This was linked to difficulties in determining whether the intervention status was classified retrospectively at follow-up, i.e. post hoc. The overall risk of bias ranged from moderate to critical; this was most closely linked to the assessment of confounders.

Conclusion

The ROBINS-I tool is a conceptually rigorous tool which focusses on risk of bias due to the counterfactual. Difficulties in applying ROBINS-I may be due to poor design and reporting of evaluations of natural experiments. While the quality of reporting may improve in the future, improved guidance on applying ROBINS-I is needed to enable existing evidence from natural experiments to be assessed appropriately and consistently. We hope future refinements to ROBINS-I will address some of the issues raised here to allow wider use of the tool.
Appendix
Available only for authorised users
Literature
1.
Reeves BC, Deeks JJ, Higgins JP, Wells GA, on behalf of the Cochrane Non-Randomised Studies Methods Group: Including non-randomised studies (chapter 13). In: Cochrane handbook for systematic reviews of interventions version 510 [updated March 2011] (available from wwwcochrane-handbookorg). Higgins J. Green Se: Wiley; 2011.
2.
Craig P, Katikireddi SV, Leyland A, Popham F. Natural experiments: an overview of methods, approaches, and contributions to public health intervention research. Annu Rev Public Health. 2017;38(1):39–56.CrossRefPubMed
3.
Katikireddi SV, Egan M, Petticrew M. How do systematic reviews incorporate risk of bias assessments into the synthesis of evidence? A methodological study. J Epidemiol Community Health. 2015;69(2):189–95.CrossRefPubMed
4.
Sanderson S, Tatt ID, Higgins JPT. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol. 2007;36(3):666–76.CrossRefPubMed
5.
Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, Henry D, Altman DG, Ansari MT, Boutron I, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355
6.
Higgins J, Sterne J, Savović J, Page M, Hróbjartsson A, Boutron I, Reeves B, Eldridge S: A revised tool for assessing risk of bias in randomized trials (RoB v2.0). In: Cochrane methods Cochrane database of systematic reviews issue 10 (Suppl 1) dxdoiorg/101002/14651858CD201601 Chandler J MJ, Boutron I, Welch V; 2016.
7.
Thomson H, Campbell M, Craig P, Hilton-Boon M, Katikireddi SV. ACROBAT-NRSi for public health: reporting on feasibility & utility of applying ACROBAT to studies of housing improvement. In: Annual Cochrane colloquium: 2015. Vienna: Wiley; 2015.
8.
Morgan R, Thayer K, Guyatt G, Blain R, Eftim S, Ross P, Santesso N, Holloway A, Schünemann H. Assessing the usability of ACROBAT-NRSI for studies of exposure and intervention in environmental health research. In: Annual Cochrane colloquium, vol. 2015. Vienna: Wiley; 2015.
9.
Couto E, Pike E, Torkilseng E, Klemp M: Inter-rater reliability of the risk of bias assessment tool: for non-randomized studies of interventions (ACROBAT-NRSI) in: Annual Cochrane Colloquium: 2015; Vienna: Wiley; 2015.
10.
Benton JS, Anderson J, Hunter RF, French DP. The effect of changing the built environment on physical activity: a quantitative review of the risk of bias in natural experiments. Int J Behav Nutr Phys Act. 2016;13(1):107.CrossRefPubMedPubMedCentral
11.
Thomson H, Thomas S, Sellstrom E, Petticrew M. Housing improvements for health and associated socio-economic outcomes [review]. Cochrane Database Syst Rev. 2013;(Issue 2):Art. No.: CD008657. 008610.001002/14651858.CD14008657.pub14651852
12.
Braubach M, Heinen D, Dame J: Preliminary results of the WHO Frankfurt housing intervention project. In. Copenhagen: World Health Organisation; 2008.
13.
Shortt N, Rugkasa J. “The walls were so damp and cold” fuel poverty and ill health in Northern Ireland: results from a housing intervention. Health & Place Part Special Issue: Environmental Justice, Population Health, Critical Theory and GIS. 2007;13(1):99–110.
14.
Hopton J, Hunt S. The health effects of improvements to housing: a longitudinal study. Hous Stud. 1996;11(2):271–86.CrossRef
15.
Somerville M, Mackenzie I, Owen P, Miles D. Housing and health: does installing heating in their homes improve the health of children with asthma? Public Health. 2000;114(6):434–9.PubMed
16.
Walker J, Mitchell R, Petticrew M, Platt S. The effects on health of a publicly funded domestic heating programme: a prospective controlled study. J Epidemiol Community Health. 2009;63(1):12–7.CrossRefPubMed
17.
Thomson H, Thomas S, Sellstrom E, Petticrew M. The health impacts of housing improvement: a systematic review of intervention studies from 1887 to 2007. Am J Public Health. 2009;99(S3):S681–92.CrossRefPubMedPubMedCentral
18.
19.
Armstrong R, Waters E, Doyle J: Reviews in public health and health promotion (chapter 21). In: Cochrane handbook for systematic reviews of interventions version 510 [updated March 2011] (Available from wwwcochrane-handbookorg). Edited by Higgins J, Green S: Wiley; 2011.
20.
Sterne J, Higgins Julian PT, Elbers R, Reeves B, and the development group for ROBINS-I: Risk of bias in non-randomized studies of interventions (ROBINS-I): detailed guidance, updated 12 October 2016. Available from http://​www.​riskofbias.​info [accessed 1/2/17]. In.
21.
22.
Craig P, Cooper C, Gunnell D, Haw S, Lawson KD, Macintyre S, Ogilvie D, Petticrew M, Reeves BC, Sutton M, et al. Using natural experiments to evaluate population health interventions. London: Medical Research Council; 2012.
23.
Armijo-Olivo S, Ospina M, da Costa BR, Egger M, Saltaji H, Fuentes J, Ha C, Cummings GG. Poor reliability between Cochrane reviewers and blinded external reviewers when applying the Cochrane risk of bias tool in physical therapy trials. PLoS One. 2014;9(5):e96920.CrossRefPubMedPubMedCentral
24.
Armijo-Olivo S, Stiles CR, Hagen NA, Biondo PD, Cummings GG: Assessment of study quality for systematic reviews: a comparison of the Cochrane collaboration risk of bias tool and the effective public health practice project quality assessment tool: methodological research. J Eval Clin Pract. 2012;18(1):12–8.
25.
Voss PH, Rehfuess EA. Quality appraisal in systematic reviews of public health interventions: an empirical study on the impact of choice of tool on meta-analysis. J Epidemiol Community Health. 2013;67(1):98–104.CrossRefPubMed
26.
Hartling L, Hamm MP, Milne A, Vandermeer B, Santaguida PL, Ansari M, Tsertsvadze A, Hempel S, Shekelle P, Dryden DM. Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. J Clin Epidemiol. 2013;66(9):973–81.CrossRefPubMed
27.
Hartling L, Milne A, Hamm MP, Vandermeer B, Ansari M, Tsertsvadze A, Dryden DM. Testing the Newcastle Ottawa scale showed low reliability between individual reviewers. J Clin Epidemiol. 2013;66(9):982–93.CrossRefPubMed
28.
Hartling L, Ospina M, Liang Y, Dryden DM, Hooton N, Seida JK, Klassen TP. Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ. 2009;339
29.
Armstrong R, Campbell M, Craig P, Hoffmann T, Katikireddi SV, Waters E. Reporting guidelines for population health and policy interventions: TIDieR-PHP. Lancet. 2016;386:S19.CrossRef
30.
Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, Altman DG, Barbour V, Macdonald H, Johnston M, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ British Medical Journal. 2014;348
31.
Humphreys DK, Panter J, Ogilvie D. Questioning the application of risk of bias tools in appraising evidence from natural experimental studies: critical reflections on Benton et al., IJBNPA 2016. Int J Behav Nutr Phys Act. 2017;14(1):49.CrossRefPubMedPubMedCentral
Metadata
Title
Applying the ROBINS-I tool to natural experiments: an example from public health
Authors
Hilary Thomson
Peter Craig
Michele Hilton-Boon
Mhairi Campbell
Srinivasa Vittal Katikireddi
Publication date
01-12-2018
Publisher
BioMed Central
Published in
Systematic Reviews / Issue 1/2018
Electronic ISSN: 2046-4053
DOI
https://doi.org/10.1186/s13643-017-0659-4

Other articles of this Issue 1/2018

Systematic Reviews 1/2018 Go to the issue

Protocol

A protocol for a network meta-analysis of interventions to treat patients with sudden sensorineural hearing loss

Protocol

Quality of life of women who practice dance: a systematic review protocol

Research

Resting blood pressure reductions following handgrip exercise training and the impact of age and sex: a systematic review and narrative synthesis

Protocol

The use of standardized management protocols for critically ill patients with non-traumatic subarachnoid hemorrhage: a protocol of a systematic review and meta-analysis

Research

What are the barriers and facilitators for third sector organisations (non-profits) to evaluate their services? A systematic review

Protocol

Body image dissatisfaction in patients with inflammatory bowel disease: a systematic review protocol