Top

Systematic Reviews

Published in:

Open Access 01-12-2022 | Commentary

The Campbell Collaboration’s systematic review of school-based anti-bullying interventions does not meet mandatory methodological standards

Authors: Julia H. Littell, Dennis M. Gorman

Published in: Systematic Reviews | Issue 1/2022

Abstract

Background

Many published reviews do not meet the widely accepted PRISMA standards for systematic reviews and meta-analysis. Campbell Collaboration and Cochrane reviews are expected to meet even more rigorous standards, but their adherence to these standards is uneven. For example, a newly updated Campbell systematic review of school-based anti-bullying interventions does not appear to meet many of the Campbell Collaboration’s mandatory methodological standards.

Issues

In this commentary, we document methodological problems in the Campbell Collaboration's new school-based anti-bullying interventions review, including (1) unexplained deviations from the protocol; (2) inadequate documentation of search strategies; (3) inconsistent reports on the number of included studies; (4) undocumented risk of bias ratings; (5) assessments of selective outcome reporting bias that are not transparent, not replicable, and appear to systematically underestimate risk of bias; (6) unreliable assessments of risk of publication bias; (7) use of a composite scale that conflates distinct risks of bias; and (8) failure to consider issues related to the strength of the evidence and risks of bias in interpreting results and drawing conclusions. Readers who are unaware of these problems may place more confidence in this review than is warranted. Campbell Collaboration editors declined to publish our comments and declined to issue a public statement of concern about this review.

Conclusions

Systematic reviews are expected to use transparent methods and follow relevant methodological standards. Readers should be concerned when these expectations are not met, because transparency and rigor enhance the trustworthiness of results and conclusions. In the tradition of Donald T. Campbell, there is need for more public debate about the methods and conclusions of systematic reviews, and greater clarity regarding applications of (and adherence to) published standards for systematic reviews.

Page MJ, Shamseer L, Altman DG, Tetzlaff J, Sampson M, Tricco AC, et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. Low N, editor. PLoS Med. 2016;13(5):e1002028.CrossRef

Wang X, Welch V, Li M, Yao L, Littell J, Li H, et al. The methodological and reporting characteristics of Campbell reviews: a systematic review. Campbell Syst Rev. 2021;17(1):e1134.

The Campbell Collaboration. Campbell systematic reviews: policies and guidelines, Version 1.7, page 5. 2020. Available from: https://onlinelibrary.wiley.com/page/journal/18911803/homepage/author-guidelines. Accessed 10 Mar 2022.

The Methods Coordinating Group of the Campbell Collaboration. Methodological expectations of Campbell Collaboration intervention reviews: conduct standards. The Campbell Collaboration; 2019 Oct. Available from: https://campbellcollaboration.org/library/campbell-methods-conduct-standards.html. Accessed 10 Mar 2022.

The Methods Coordinating Group of the Campbell Collaboration. Methodological expectations of Campbell Collaboration intervention reviews: reporting standards. The Campbell Collaboration; 2019 Oct. Available from: https://campbellcollaboration.org/library/campbell-methods-reporting-standards.html. Accessed 10 Mar 2022.

Gaffney H, Ttofi MM, Farrington DP. Effectiveness of school-based programs to reduce bullying perpetration and victimization: an updated systematic review and meta-analysis. Campbell Syst Rev. 2021;17(2). https://doi.org/10.1002/cl2.1143.

Kirkham JJ, Dwan KM, Blümle A, von Elm E, Williamson PR. How much participant outcome data Is missing from sight: Findings from a cohort of trials submitted to a German research ethics committee. PLoS One. 2016;11(6):e0157883.CrossRef

Pigott TD, Valentine JC, Polanin JR, Williams RT, Canada DD. Outcome-reporting bias in education research. Educ Res. 2013;42(8):424–32.CrossRef

Dwan K, Gamble C, Williamson PR, Kirkham JJ, the Reporting Bias Group. Systematic review of the empirical evidence of study publication bias and outcome reporting bias — an updated review. PLoS One. 2013;8(7):e66844.CrossRef

10.

Schriger DL, Savage DF, Altman DG. Presentation of continuous outcomes in randomised trials: an observational study. BMJ. 2012;345(Dec18 2):e8486.

11.

Song F, Parekh S, Hooper L, Loke Y, Ryder J, Sutton A, et al. Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess. 2010;14(8) Available from: https://www.journalslibrary.nihr.ac.uk/hta/hta14080/. Accessed 11 Mar 2022.

12.

Taylor NJ, Gorman DM. Registration and primary outcome reporting in behavioral health trials. BMC Med Res Methodol. 2022;22(1):41.CrossRef

13.

Bradley HA, Rucklidge JJ, Mulder RT. A systematic review of trial registration and selective outcome reporting in psychotherapy randomized controlled trials. Acta Psychiatr Scand. 2017;135(1):65–77.CrossRef

14.

Azar M, Riehm KE, McKay D, Thombs BD. Transparency of outcome reporting and trial registration of randomized controlled trials published in the journal of consulting and clinical psychology. Schooling CM, editor. PLoS One. 2015;10(11):e0142894.CrossRef

15.

Milette K, Roseman M, Thombs BD. Transparency of outcome reporting and trial registration of randomized controlled trials in top psychosomatic and behavioral health journals: A systematic review. J Psychosom Res. 2011;70(3):205–17.CrossRef

16.

Gaffney H, Farrington DP, White H. Anti-bullying programmes: Toolkit technical report. UK: Youth Endowment Fund; 2021. p. 58.

17.

Gaffney H, Farrington DP, Espelage DL, Ttofi MM. Are cyberbullying intervention and prevention programs effective? A systematic and meta-analytical review. Aggress Violent Behav. 2019;45:134–53.CrossRef

18.

Gaffney H, Farrington DP, Ttofi MM. Examining the Effectiveness of School-Bullying Intervention Programs Globally: a Meta-analysis. Int J Bullying Prev. 2019;1(1):14–31.CrossRef

19.

Gaffney H, Ttofi MM, Farrington DP. Evaluating the effectiveness of school-bullying prevention programs: An updated meta-analytical review. Aggress Violent Behav. 2019;45:111–33.CrossRef

20.

Gaffney H, Ttofi MM, Farrington DP. What works in anti-bullying programs? Analysis of effective intervention components. J Sch Psychol. 2021;85:37–56.CrossRef

21.

Farrington DP, Ttofi MM. School-based programs to reduce bullying and victimization. Campbell Syst Rev. 2009;5(1). https://doi.org/10.4073/csr.2009.6.

22.

Farrington DP, Baldry AC, Kyvsgaard B, Ttofi MM. PROTOCOL: effectiveness of programs to prevent school bullying. Campbell Syst Rev. 2008;4(1):1–29.CrossRef

23.

Yanagida T, Strohmeier D, Spiel C. Dynamic change of aggressive behavior and victimization among adolescents: effectiveness of the ViSC program. J Clin Child Adolesc Psychol. 2019;48(sup1):S90–104.CrossRef

24.

Wölfer R, Scheithauer H. Social influence and bullying behavior: intervention-based network dynamics of the fairplayer.manual bullying prevention program: Social Influence and Bullying. Aggress Behav. 2014;40(4):309–19.CrossRef

25.

Baldry AC, Farrington DP. Evaluation of an intervention program for the reduction of bullying and victimization in schools. Aggr Behav. 2004;30(1):1–15.

26.

Beran T, Shapiro B. Evaluation of an anti-bullying program: student reports of knowledge and confidence to manage bullying. Can J Educ / Revue canadienne de l’éducation. 2005;28(4):700.

27.

Berry K, Hunt CJ. Evaluation of an intervention program for anxious adolescent boys who are bullied at school. J Adolesc Health. 2009;45(4):376–82.

28.

Bonell C, Fletcher A, Fitzgerald-Yau N, Hale D, Allen E, Elbourne D, et al. Initiating change locally in bullying and aggression through the school environment (INCLUSIVE): a pilot randomised controlled trial. Health Technol Assess. 2015;19(53):1–110.CrossRef

29.

Boulton MJ, Flemington I. The effects of a short video intervention on secondary school pupils’ involvement in definitions of and attitudes towards bullying. School Psychol Int. 1996;17(4):331–45.

30.

Brown EC, Low S, Smith BH, Haggerty KP. Outcomes from a school-randomized controlled trial of steps to respect: a bullying prevention program. 2011;40(3):423–43.

31.

Chaux E, Velásquez AM, Schultze-Krumbholz A, Scheithauer H. Effects of the cyberbullying prevention program media heroes (Medienhelden) on traditional bullying: effects of media heroes on traditional bullying. Aggr Behav. 2016;42(2):157–65.

32.

Cissner AB, Ayoub LH. Building healthy teen relationships: an evaluation of the fourth R curriculum with middle school students in the Bronx. USA: Center for Court Innovation; 2014.

33.

Connolly J, Josephson W, Schnoll J, Simkins-Strong E, Pepler D, MacPherson A, et al. Evaluation of a youth-led program for preventing bullying, sexual harassment, and dating aggression in middle schools. J Early Adolesc. 2015;35(3):403–34.

34.

Cross D, Monks H, Hall M, Shaw T, Pintabona Y, Erceg E, et al. Three‐year results of the Friendly Schools whole-of-school intervention on children’s bullying behaviour. Br Educ Res J. 2011;37(1):105–29.

35.

DeRosier ME, Marcus SR. Building friendships and combating bullying: effectiveness of S.S.GRIN at one-year follow-up. J Clin Child Adolesc Psychol. 2005;34(1):140–50.

36.

Domino M. Measuring the impact of an alternative approach to school bullying. J School Health. 2013;83(6):430–7.

37.

Espelage DL, Rose CA, Polanin JR. Social-emotional learning program to reduce bullying, fighting, and victimization among middle school students with disabilities. Remedial Spec Educ. 2015;36(5):299–311.

38.

Fekkes M, Pijpers FIM, Verloove-Vanhorick SP. Effects of antibullying school program on bullying and health complaints. Arch Pediatr Adolesc Med. 2006;160(6):638.

39.

Fekkes M, van de Sande MCE, Gravesteijn JC, Pannebakker FD, Buijs GJ, Diekstra RFW, et al. Effects of the Dutch Skills for Life program on the health behavior, bullying, and suicidal ideation of secondary school students. Health Educ. 2016;116(1):2–15.

40.

Fonagy P, Twemlow SW, Vernberg EM, Nelson JM, Dill EJ, Little TD, et al. A cluster randomized controlled trial of child-focused psychiatric consultation and a school systems-focused intervention to reduce aggression. J Child Psychol Psychiatry. 2009;50(5):607–16.

41.

Frey KS, Hirschstein MK, Snell JL, Edstrom LVS, MacKenzie EP, Broderick CJ. Reducing playground bullying and supporting beliefs: an experimental trial of the steps to respect program. Dev Psychol. 2005;41(3):479–90.

42.

Garaigordobil M, Martínez-Valderrey V. Effects of Cyberprogram 2.0 on “face-to-face” bullying, cyberbullying, and empathy. Psicothema. 2015;(27.1):45–51.

43.

Holen S, Waaktaar T, Lervåg A, Ystgaard M. Implementing a universal stress management program for young school children: are there classroom climate or academic effects? Scand J Educ Res. 2013;57(4):420–44.

44.

Hunt C. The Effect of an Education Program on Attitudes and Beliefs about Bullying and Bullying Behaviour in Junior Secondary School Students. Child Adolesc Ment Health. 2007;12(1):21–6.CrossRef

45.

Jenson JM, Brisson D, Bender KA, Williford AP. Effects of the youth matters prevention program on patterns of bullying and victimization in elementary and middle school. Soc Work Res. 2013;37(4):361–72.

46.

Ju Y, Wang S, Zhang W. Intervention research on school bullying in primary schools. Front Educ China. 2009;4(1):111–22.CrossRef

47.

Kaljee L, Zhang L, Langhaug L, Munjile K, Tembo S, Menon A, et al. A randomized-control trial for the teachers’ diploma programme on psychosocial care, support and protection in Zambian government primary schools. Psychol Health Med. 2017;22(4):381–92.

48.

Kärnä A, Voeten M, Little TD, Poskiparta E, Kaljonen A, Salmivalli C. A large-scale evaluation of the KiVa antibullying program: grades 4-6: evaluation of KiVa antibullying program. Child Dev. 2011b;82(1):311–30.

49.

Kärnä A, Voeten M, Little TD, Alanen E, Poskiparta E, Salmivalli C. Effectiveness of the KiVa Antibullying Program: Grades 1–3 and 7–9. J Educ Psychol. 2013;105(2):535–51.

50.

Knowler C, Frederickson N. Effects of an emotional literacy intervention for students identified with bullying behaviour. Educ Psychol. 2013;33(7):862–83.

51.

Krueger LM. The implementation of an anti-bullying program to reduce bullying behaviors on elementary school buses [Internet] [Ed.D.]. ProQuest Dissertations and Theses. Ann Arbor: D’Youville College; 2010.

52.

Li KK, Washburn I, DuBois DL, Vuchinich S, Ji P, Brechling V, et al. Effects of the Positive Action programme on problem behaviours in elementary school students: A matched-pair randomised control trial in Chicago. Psychol Health. 2011;26(2):187–204.

53.

McLaughlin LP. The effect of cognitive behavioral therapy and cognitive behavioral therapy plus media on the reduction of bullying and victimization and the increase of empathy and bystander response in a bully prevention program for urban sixth -grade students [Internet] [Ph.D.]. ProQuest Dissertations and Theses. Ann Arbor: The University of Toledo; 2009.

54.

Meyer N, Lesch E. An analysis of the limitations of a behavioural programme for bullying boys from a subeconomic environment. null. 2000;12(1):59–69.

55.

Nocentini A, Menesini E. KiVa anti-bullying program in Italy: evidence of effectiveness in a randomized control trial. Prev Sci. 2016;17(8):1012–23.

56.

Ostrov JM, Godleski SA, Kamper-DeMarco KE, Blakely-McClure SJ, Celenza L. Replication and extension of the early childhood friendship project: effects on physical and relational bullying. Stormont M, editor. School Psychol Rev. 2015;44(4):445–63.CrossRef

57.

Polanin MK. [Ph.D.]. ProQuest Dissertations and Theses. [Ann Arbor]: Loyola University Chicago; 2014. Effects of cultural awareness training in conjunction with an established bullying prevention program [Internet].

58.

Rosenbluth B, Whitaker DJ, Sanchez E, Valle LA. The Expect Respect project: preventing bullying and sexual harassment in US elementary schools. In: Pepler D, Rigby K, Smith PK, editors. Bullying in Schools: How Successful Can Interventions Be? Cambridge: Cambridge University Press; 2004. p. 211–34.

59.

Spröber N, Schlottke PF, Hautzinger M. ProACT + E: Ein Programm zur Prävention von “bullying“ an Schulen und zur Förderung der positiven Entwicklung von Schülern. Zeitschrift für Klinische Psychologie und Psychotherapie. 2006;35(2):140–50.

60.

Stallard P, Phillips R, Montgomery A, Spears M, Anderson R, Taylor J, et al. A cluster randomised controlled trial to determine the clinical effectiveness and cost-effectiveness of classroom-based cognitive–behavioural therapy (CBT) in reducing symptoms of depression in high-risk adolescents. Health Technol Assess. 2013;17(47). https://doi.org/10.3310/hta17470.

61.

Topper L. Bullying victimisation and alcohol-misuse in adolescence: Investigating the functional relationship and new prevention strategies [Doctoral dissertation]. King’s College London (University of London); 2012.

62.

Trip S, Bora C, Sipos-Gug S, Tocai I, Gradinger P, Yanagida T, et al. Bullying prevention in schools by targeting cognitions, emotions, and behavior: Evaluating the effectiveness of the REBE-ViSC program. J Couns Psychol. 2015;62(4):732–40.

63.

Tsiantis ACJ, Beratis IN, Syngelaki EM, Stefanakou A, Asimopoulos C, Sideridis GD, et al. The effects of a clinical prevention program on bullying, victimization, and attitudes toward school of elementary school students. Behav Disord. 2013;38(4):243–57.

64.

Waasdorp TE. The impact of schoolwide positive behavioral interventions and supports on bullying and peer rejection: a randomized controlled effectiveness trial. Arch Pediatr Adolesc Med. 2012;166(2):149.

65.

Stallard P. A single blind randomised controlled trial to determine the effectiveness of group Cognitive Behaviour Therapy (CBT) in the prevention of depression in high risk adolescents. ISRCTN. Available from: http://www.isrctn.com/ISRCTN19083628. Accessed 10 Mar 2022.

66.

Stallard P, Montgomery AA, Araya R, Anderson R, Lewis G, Sayal K, et al. Protocol for a randomised controlled trial of a school based cognitive behaviour therapy (CBT) intervention to prevent depression in high risk adolescents (PROMISE). Trials. 2010;11(1):114.CrossRef

67.

Viner R. INitiating Change Locally in bUllyIng and aggression through the School EnVironment. ISRCTN; 2011. Available from: http://www.isrctn.com/ISRCTN885270781. Accessed 10 Mar 2022.

68.

Norris SL, Holmer HK, Ogden LA, Fu R, Abou-Setta AM, Viswanathan MS, et al. Selective outcome reporting as a source of bias in reviews of comparative effectiveness. Rockville, MD: Agency for Healtcare Research and Quality; 2012 Aug. Report No.: AHRQ Publication No. 12-EHC110-EF. Available from: www.effectivehealthcare.ahrq.gov/reports/final.cfm. Accessed 11 Mar 2022.

69.

Lau J, Ioannidis JPA, Terrin N, Schmid CH, Olkin I. The case of the misleading funnel plot. BMJ. 2006;333(7568):597–600.CrossRef

70.

Terrin N, Schmid CH, Lau J. In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. J Clin Epidemiol. 2005;58:894–901.CrossRef

71.

Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L, Moreno SG. Assessing publication bias in meta-analyses in the presence of between-study heterogeneity. J R Stat Soc A Stat Soc. 2010;173(3):575–91.CrossRef

72.

Terrin N, Schmid CH, Lau J, Olkin I. Adjusting for publication bias in the presence of heterogeneity. Stat Med. 2003;22(13):2113–26. 1.CrossRef

73.

Shi L, Lin L. The trim-and-fill method for publication bias: practical guidelines and recommendations based on a large database of meta-analyses. Medicine. 2019;98(23) Available from: https://journals.lww.com/md-journal/Fulltext/2019/06070/The_trim_and_fill_method_for_publication_bias_.70.aspx.

74.

Campbell DT. The experimenting society. In: Dunn WN, editor. The experimenting society: essays in honor of Donald T Campbell: Transaction Publishers; 1998. p. 35–68.

Title: The Campbell Collaboration’s systematic review of school-based anti-bullying interventions does not meet mandatory methodological standards
Authors: Julia H. Littell
Dennis M. Gorman
Publication date: 01-12-2022
Publisher: BioMed Central
Published in: Systematic Reviews / Issue 1/2022
Electronic ISSN: 2046-4053
DOI: https://doi.org/10.1186/s13643-022-01998-1

Keynote webinar | Spotlight on medication adherence

Springer Medicine

The Campbell Collaboration’s systematic review of school-based anti-bullying interventions does not meet mandatory methodological standards

Abstract

Background

Issues

Conclusions

Keynote webinar | Spotlight on medication adherence

Springer Medicine

Abstract

Background

Issues

Conclusions

Please log in to get access to this content

Other articles of this Issue 1/2022

Assessing the causal relationship between income inequality and mortality and self-rated health: protocol for systematic review and meta-analysis

What is the potential for plural ownership to support a more inclusive economy? A systematic review protocol

Investigating the safety and efficacy of hematopoietic and mesenchymal stem cell transplantation for treatment of T1DM: a systematic review and meta-analysis

The evidence synthesis and meta-analysis in R conference (ESMARConf): levelling the playing field of conference accessibility and equitability

What is the most appropriate method for the measurement of the range of motion in the lumbar spine and how does surgical fixation affect the range of movement of the lumbar spine in adolescent idiopathic scoliosis? A systematic review protocol

Postoperative pain and pain management and neurocognitive outcomes after non-cardiac surgery: a protocol for a series of systematic reviews