Skip to main content
Key Specialties
Cardiology Diabetology Endocrinology Gastroenterology Geriatrics and Gerontology Gynecology and Obstetrics Hematology Infectious Disease Internal Medicine Nephrology Neurology Oncology Pediatrics Respiratory Medicine Rheumatology Surgery
Congress Coverage
2024 ACC Congress 2023 ESMO Congress 2023 EASD Congress 2023 ERS Congress 2023 ESC Congress
Clinical Collections
Advances in Alzheimer’s

At a glance: The STEP trials

A round-up of the STEP phase 3 clinical trials evaluating semaglutide for weight loss in people with overweight or obesity.
ADVANCED SEARCH
Log in
Top
BMC Medical Research Methodology
Published in: BMC Medical Research Methodology 1/2012

Open Access 01-12-2012 | Research article

Systematic reviews need to consider applicability to disadvantaged populations: inter-rater agreement for a health equity plausibility algorithm

Authors: Vivian Welch, Kevin Brand, Elizabeth Kristjansson, Janet Smylie, George Wells, Peter Tugwell

Published in: BMC Medical Research Methodology | Issue 1/2012

Login to get access

Abstract

Background

Systematic reviews have been challenged to consider effects on disadvantaged groups. A priori specification of subgroup analyses is recommended to increase the credibility of these analyses. This study aimed to develop and assess inter-rater agreement for an algorithm for systematic review authors to predict whether differences in effect measures are likely for disadvantaged populations relative to advantaged populations (only relative effect measures were addressed).

Methods

A health equity plausibility algorithm was developed using clinimetric methods with three items based on literature review, key informant interviews and methodology studies. The three items dealt with the plausibility of differences in relative effects across sex or socioeconomic status (SES) due to: 1) patient characteristics; 2) intervention delivery (i.e., implementation); and 3) comparators. Thirty-five respondents (consisting of clinicians, methodologists and research users) assessed the likelihood of differences across sex and SES for ten systematic reviews with these questions. We assessed inter-rater reliability using Fleiss multi-rater kappa.

Results

The proportion agreement was 66% for patient characteristics (95% confidence interval: 61%-71%), 67% for intervention delivery (95% confidence interval: 62% to 72%) and 55% for the comparator (95% confidence interval: 50% to 60%). Inter-rater kappa, assessed with Fleiss kappa, ranged from 0 to 0.199, representing very low agreement beyond chance.

Conclusions

Users of systematic reviews rated that important differences in relative effects across sex and socioeconomic status were plausible for a range of individual and population-level interventions. However, there was very low inter-rater agreement for these assessments. There is an unmet need for discussion of plausibility of differential effects in systematic reviews. Increased consideration of external validity and applicability to different populations and settings is warranted in systematic reviews to meet this need.
Appendix
Available only for authorised users
Literature
1.
Whitehead M: The concepts and principles of equity and health. Int J Health Serv. 1992, 22: 429-445. 10.2190/986L-LHQ6-2VTE-YRRN.CrossRefPubMed
2.
Kelly MP, Morgan A, Exworthy M, Popay J, Tugwell P, Robinson V, Simpson S, Narayan T, Myer L, Houweling T, et al: The social determinants of health: developing an evidence base for political action: final report to the world health organization commission on the social determinants of health. 2007, Geneva
3.
White M, Adams J, Heywood P: How and why do interventions that increase health overall widen inequalities with populations?. Social inequality and public health. Edited by: Babones SJ. 2009, Bristol: The Policy Press, University of Bristol, 65-82. 1
4.
World Health Assembly: Resolution 5834 on the ministerial summit on health research. 2005, 5834-
5.
Lavis JN, Davies HTO, Gruen RL: Working within and beyond the cochrane collaboration to make systematic reviews more useful to healthcare managers and policy makers. Healthcare Policy. 2006, 1: 21-33.PubMedPubMedCentral
6.
Mackenbach JP: Tackling inequalities in health: the need for building a systematic evidence base. J Epidemiol Community Health. 2003, 57: 162-10.1136/jech.57.3.162.CrossRefPubMedPubMedCentral
7.
Thomas S, Fayter D, Misso K, Ogilvie D, Petticrew M, Sowden A, Whitehead M, Worthy G, Thomas S, Fayter D, et al: Population tobacco control interventions and their effects on social inequalities in smoking: systematic review. [Review] [109 refs]. Tob Control. 2008, 17: 230-237. 10.1136/tc.2007.023911.CrossRefPubMedPubMedCentral
8.
Chopra M, Munro S, Lavis JN, Vist G, Bennett S: Effects of policy options for human resources for health: an analysis of systematic reviews. Lancet. 2008, 371: 668-774. 10.1016/S0140-6736(08)60305-0.CrossRefPubMed
9.
Tsikata S, Robinson V, Petticrew M, Kristjansson B, Moher D, McGowan J, Shea B, Wells G, Tugwell P: Do cochrane systematic reviews contain useful information about health equity? [abstract]. XI cochrane colloquium: evidence, health care and culture; 2003 Oct 26 31. 2003, Barcelona, Spain, 77-
10.
Petticrew M, Whitehead M, Macintye SJ, Graham H, Egan M: Evidence for public health policy on inequalities: 1: the reality according to policymakers. J Epidemiol Community Health. 2004, 58: 811-816. 10.1136/jech.2003.015289.CrossRefPubMedPubMedCentral
11.
Lavis JN, Davies H, Oxman A, Denis JL, Golden-Biddle K, Ferlie E: Towards systematic reviews that inform health care management and policy-making. J Health Serv Res Policy. 2005, 10: 35-48. 10.1258/1355819054308549.CrossRefPubMed
12.
Tugwell P, Petticrew M, Robinson V, Kristjansson E, Maxwell L: Cochrane and campbell collaborations, and health equity. Lancet. 2006, 367: 1128-1130. 10.1016/S0140-6736(06)68490-0.CrossRefPubMed
13.
Petticrew M, Tugwell P, Welch V, Ueffing E, Kristjansson E, Armstrong R, Doyle J, Waters E: Better evidence about wicked issues in tackling health inequities. J Public Health. 2009, 31: 453-456. 10.1093/pubmed/fdp076.CrossRef
14.
Evans T, Brown H: Road traffic crashes: operationalizing equity in the context of health sector reform. Inj Control Saf Promot. 2003, 10: 11-12. 10.1076/icsp.10.1.11.14117.CrossRefPubMed
15.
Kavanagh J, Oliver S, Lorenc T: Reflections in developing and using PROGRESS-plus. Equity update (2). 2008, Ottawa, 1-3.
16.
US Department of Health and Human Services: Understanding and improving health. 2000, Washington, DC
17.
Marmot M, Friel S, Bell R, Houweling TA, Taylor S: Commission on social determinants of health: closing the gap in a generation: health equity through action on the social determinants of health. Lancet. 2008, 372: 1661-1669. 10.1016/S0140-6736(08)61690-6.CrossRefPubMed
18.
Sun X, Briel M, Walter SD, Guyatt GH: Is a subgroup effect believable? Updating criteria to evaluate the credibility of subgroup analyses. Br Med J. 2010, 340: C117-10.1136/bmj.c117.CrossRef
19.
Sun X, Briel M, Busse JW, You JJ, Akl EA, Mejza F, Bala MM, Bassler D, Mertz D, Diaz-Granados N, Vandvik PO, Malaga G, Srinathan SK, Dahm P, Johnston BC, Alonso-Coello P, Hassouneh B, Walter SD, Heels-Ansdell D, Bhatnagar N, Altman DG, Guyatt GH: Credibility of claims of subgroup effects in randomised controlled trials: systematic review. Br Med J. 2012, 344: e1553-10.1136/bmj.e1553.CrossRef
20.
Petticrew M, Tugwell P, Kristjansson E, Oliver S, Ueffing E, Welch V: Damned if you do, damned if you don't: subgroup analysis and equity. J Epidemiol Community Health. 2011, epub ahead of print
21.
Feinstein AR: Clinimetrics. 1987, New Haven: Yale University Press
22.
Feinstein AR: Feinstein AR. Clinical epidemiology: the architecture of clinical research. 1985, Philadelphia: Saunders, 39-52. Philadelphia: Saunders; 1985
23.
Dans AL, Dans LF, Guyatt GH, Richardson S: Users' Guides to the medical literature: XIV. How to decide on the applicability of clinical trial results to your patient. Evidence-based medicine working group.[see comment]. JAMA. 1998, 279: 545-549. 10.1001/jama.279.7.545.CrossRefPubMed
24.
Greenaway C, Sandoe A, Vissandjee B, Kitai I, Gruner D, Ueffing E, Wobeser W, Menzies D, Schwartzman K: Tuberculosis: evidence review for newly arriving immigrants. Can Med Assoc J. 2010, 183 (12): E939-E951.CrossRef
25.
Welch V, Tugwell P, Wells GA, Kristjansson E, Petticrew M, McGowan J, DeMontigny J, Benkhalti M, Ueffing E: How effects on health equity are assessed in systematic reviews of interventions. Cochrane Database Syst Rev. 2010, MR000028-12
26.
Driedger SM, Gallois C, Sanders CB, Santesso N, Driedger SM, Gallois C, Sanders CB, Santesso N: Finding common ground in team-based qualitative research using the convergent interviewing method. Qual Health Res. 2006, 16: 1145-1157. 10.1177/1049732306289705.CrossRefPubMed
27.
Welch V, Smylie JK, Kristjansson E, Brand K, Tugwell P, Wells GA: What is the role of systematic reviews in tackling health inequity?. 2010, University of Ottawa, PhD Thesis
28.
Dans AL, Dans LF, Guyatt G: Applying results to individual patients. Users' Guides to the medical literature: a manual for evidence-based clinical practice. Edited by: Guyatt G, Rennie D, Meade MO, Cook DJ. 2008, New York, USA: The McGraw-Hill Companies, Inc, 273-289. 2
29.
Fleiss JL: Measuring nominal scale agreement among many raters. Psychol Bull. 1971, 76: 378-382.CrossRef
30.
Cicchetti DV, Feinstein AR: High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology. 1990, 43: 551-558. 10.1016/0895-4356(90)90159-M.CrossRefPubMed
31.
Evans DB, Lim SS, Adam T, Edejer TT: WHO choosing interventions that are cost effective (CHOICE) millennium development goals team: evaluation of current strategies and future priorities for improving health in developing countries. BMJ. 2005, 331: 1457-1461. 10.1136/bmj.38658.675243.94.CrossRefPubMedPubMedCentral
32.
Jamison DT, Breman JG, Measham AR, Alleyne G, Claeson M, Evans DB, Jha P, Mills A, Musgrove P: Priorities in health. 2006, New York: Oxford University PressCrossRef
33.
Mathers CD, Loncar D: Projections of global mortality and burden of disease from 2002 to 2030. PLoS Med. 2006, 3: 3442-CrossRef
34.
Tugwell P, Boers M, Brooks P, Simon L, Strand V, Idzerda L: OMERACT: an international initiative to improve outcome measurement in rheumatology. Trials. 2007, 8: 38-10.1186/1745-6215-8-38.CrossRefPubMedPubMedCentral
35.
Vidanapathirana J, Abramson MJ, Forbes A, Fairley C: Mass media interventions for promoting HIV testing. Cochrane Database Syst Rev. 2005, CD004775-3
36.
Main C, Thomas S, Ogilvie D, Stirk L, Petticrew M, Whitehead M, Sowden A: Population tobacco control interventions and their effects on social inequalities in smoking: placing an equity lens on existing systematic reviews. BMC Publ Health. 2008, 8: 178-10.1186/1471-2458-8-178.CrossRef
37.
Bisson J, Andrew M: Psychological treatment of post-traumatic stress disorder (PTSD). Cochrane Database Syst Rev. 2007, CD003388-3
38.
Wright JM, Musini VM: First-line drugs for hypertension. Cochrane Database Syst Rev. 2009, Art. No. CD001841
39.
Riaz Y, Mehta JS, Wormald R, Evans JR, Foster A, Ravilla T, Snelligen T: Surgical interventions for age-related cataract. Cochrane Database Syst Rev. 2006, 10.1002/14651858.CD001323.pub2. Art. No. CD001323, 4CrossRef
40.
Demicheli V, Jefferson T, Rivetti A, Price D: Vaccines for measles, mumps and rubella in children. Cochrane Database Syst Rev. 10.1002/14651858.CD004407.pub2. Art. No. CD004407, 4
41.
Arroll B, Elley CR, Fishman T, Goodyear-Smith FA, Kenealy T, Blashki G, Kerse N, McGillivray S: Antidepressants versus placebo for depression in primary care. Cochrane Database Syst Rev. 2009, 10.1002/14651858.CD007954. Art. No CD007954, 3CrossRef
42.
Sinclair D, Zani B, Donegan S, Olliaro P, Garner P: Artemisinin-based combination therapy for treating uncomplicated malaria. Cochrane Database Syst Rev. 2009
43.
Shults RA, Nichols JL, Dinh-Zarr TB, Sleet DA, Elder RW: Effectiveness of primary enforcement safety belt laws and enhanced enforcement of safety belt laws: a summary of the guide to community preventive services systematic reviews. J Safety Res. 2004, 35: 189-196. 10.1016/j.jsr.2004.03.002.CrossRefPubMed
44.
Ejemot RI, Ehiri JE, Meremikwu MM, Critchley JA: Hand washing for preventing diarrhoea. Cochrane Database Syst Rev. 2008, 10.1002/14651858.CD004265.pub2. Art No. CD004265, 1
45.
Kraemer HC, Periyakoil VS, Noda A: Tutorial in biostatistics: kappa coefficients in medical research. Stat Med. 2002, 21: 2109-2129. 10.1002/sim.1180.CrossRef
46.
Vach W: The dependence of Cohen's kappa on the prevalence does not matter. J Clin Epidemiol. 2005, 58: 655-661. 10.1016/j.jclinepi.2004.02.021.CrossRefPubMed
47.
Napoles-Springer A, Santoyo-Olsson J, O'Brien H, Stewart AL: Using cognitive interviews to develop surveys in diverse populations. Medical Care. 2006, 44: s21-s30. 10.1097/01.mlr.0000245425.65905.1d.CrossRefPubMed
48.
Tugwell P, Petticrew M, Kristjansson E, Welch V, Ueffing E, Waters E, Bonnefoy J, Morgan A, Doohan E, Kelly MP: Assessing equity in systematic reviews: realising the recommendations of the commission on social determinants of health. BMJ. 2010, 341: c4739-10.1136/bmj.c4739.CrossRefPubMed
49.
Glasziou P, Guyatt GH, Dans AL, Dans LF, Straus S, Sackett DL: Applying the results of trials and systematic reviews to individual patients.[see comment]. ACP J Club. 1998, 129: A15-A16.PubMed
50.
National Health and Medical Research Council: How to use the evidence: assessment and application of scientific evidence. 2000, Canberra, Australia
51.
Briss PA, Zaza S, Pappaioanou M, Fielding J, Wright-De AL, Truman BI, Hopkins DP, Mullen PD, Thompson RS, Woolf SH, et al: Developing an evidence-based guide to community preventive services-methods. The task force on community preventive services. Am J Prev Med. 2000, 18 (Suppl): 43-
52.
Rothwell PM: Treating individuals 2. Subgroup analysis in randomised controlled trials: importance, indications, and interpretation. Lancet. 2005, 365: 176-186. 10.1016/S0140-6736(05)17709-5.CrossRefPubMed
53.
Wang S, Moss JR, Hiller JE: Applicability and transferability of interventions in evidence-based public health. Health Promot Int. 2006, 21: 76-83.CrossRefPubMed
54.
Green LW, Glasgow RE: Evaluating the relevance, generalization, and applicability of research: issues in external validation and translation methodology. [Review] [91 refs]. Eval Health Prof. 2006, 29: 126-153. 10.1177/0163278705284445.CrossRefPubMed
55.
Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, Gøtzsche PC, Lang T: The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med. 2001, 134: 663-694.CrossRefPubMed
56.
57.
Lavis JN, Posada FB, Haines A, Osei E: Use of research to inform public policymaking.[see comment]. Lancet. 2004, 364: 1615-1621. 10.1016/S0140-6736(04)17317-0.CrossRefPubMed
58.
Dans AM, Dans L, Oxman AD, Robinson V, Acuin J, Tugwell P, Dennis R, Kang D: Assessing equity in clinical practice guidelines. [Review] [53 refs]. J Clin Epidemiol. 2007, 60: 540-546. 10.1016/j.jclinepi.2006.10.008.CrossRefPubMed
59.
Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D: The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ. 2009, 339: b2700-10.1136/bmj.b2700.CrossRefPubMedPubMedCentral
60.
EUROTHINE: final report. Tackling health inequalities in Europe: an integrated approach. 2007, Rotterdam
61.
Buendia-Rodriguez JA, Sanchez-Villamil JP, Buendia-Rodriguez JA, Sanchez-Villamil JP: Using systematic reviews for evidence-based health promotion: basic methodology issues. Revista de Salud Publica. 2006, 8 (Suppl 2): 94-105.PubMed
Metadata
Title
Systematic reviews need to consider applicability to disadvantaged populations: inter-rater agreement for a health equity plausibility algorithm
Authors
Vivian Welch
Kevin Brand
Elizabeth Kristjansson
Janet Smylie
George Wells
Peter Tugwell
Publication date
01-12-2012
Publisher
BioMed Central
Published in
BMC Medical Research Methodology / Issue 1/2012
Electronic ISSN: 1471-2288
DOI
https://doi.org/10.1186/1471-2288-12-187

Other articles of this Issue 1/2012

BMC Medical Research Methodology 1/2012 Go to the issue

Research article

Investigating linkage rates among probabilistically linked birth and hospitalization records

Research article

Patient, caregiver, health professional and researcher views and experiences of participating in research at the end of life: a critical interpretive synthesis of the literature

Research article

Forecasts of COPD mortality in Australia: 2006-2025

Research article

Reliability of self-reported health risk factors and chronic conditions questions collected using the telephone in South Australia, Australia

Technical advance

Enhanced secondary analysis of survival data: reconstructing the data from published Kaplan-Meier survival curves

Research article

Comparison of two Bayesian methods to detect mode effects between paper-based and computerized adaptive assessments: a preliminary Monte Carlo study