Top

Published in:

Open Access 01-02-2018

Quality Matters: Implementation Moderates Student Outcomes in the PATHS Curriculum

Authors: Neil Humphrey, Alexandra Barlow, Ann Lendrum

Published in: Prevention Science | Issue 2/2018

Abstract

Analyses of the relationship between levels of implementation and outcomes of school-based social and emotional learning (SEL) interventions are relatively infrequent and are typically narrowly focused. Thus, our objective was to assess the relationship between variability in a range of implementation dimensions and intervention outcomes in the Promoting Alternative Thinking Strategies (PATHS) curriculum. Implementation of PATHS was examined in 69 classrooms across 23 schools in the first year of a major randomized controlled trial. Implementation data were generated via classroom-level structured observations. In addition to factual data on dosage and reach, exploratory factor analysis of observer ratings revealed two distinct implementation dimensions, namely, “quality and participant responsiveness” and “procedural fidelity.” Student social-emotional skills, pro-social behavior, internalizing symptoms, and externalizing problems were captured through child self-report and teacher informant-report surveys (N = 1721). Hierarchical linear modeling of study data revealed that higher implementation quality and participant responsiveness was associated with significantly lower ratings of students’ externalizing problems at 12-month follow-up. Conversely, and contrary to expectations, higher dosage was associated with significantly lower pro-social behavior and social-emotional skills at 12-month follow-up. No significant associations were found between variability in either procedural fidelity or reach and any intervention outcomes. The implications of these findings are discussed, and study limitations are noted.

The rubric was designed to orient observers to behaviors indicative of the implementation indicator in question. For example, when rating students’ engagement in core activities, the explanatory notes guided observers to assess the extent to which children in the class actively participated in lesson activities (e.g., joining in role plays and answering questions).

In order to meet minimal sample size requirements (e.g., N = 10 per item), the EFA was conducted using data from the 127 observations conducted across the 2 years of the trial.

Historically, the SDQ has been scored according to a five-factor structure (emotional symptoms, conduct problems, inattention/hyperactivity, peer problems, and pro-social behavior). However, research has indicated that a three-factor structure (internalizing symptoms, externalizing problems, pro-social behavior) offers improved data fit (Goodman, Lamping & Ploubidis, 2010).

We note that one item initially designated as belonging to the fidelity dimension also loaded onto this factor; however, there was evidence that of ambiguity, with this item cross-loading onto both factors (loadings of 0.57 and 0.48 respectively).

Schonfeld et al. (2015) did not provide descriptive statistics for their implementation data and a direct comparison of overall dosage levels is therefore not possible.

Ardito, R. B., & Rabellino, D. (2011). Therapeutic alliance and outcome of psychotherapy: Historical excursus, measurements, and prospects for research. Frontiers in Psychology, 2, 1–11. doi:10.3389/fpsyg.2011.00270.CrossRef

Berkel, C., Mauricio, A. M., Schoenfelder, E., & Sandler, I. N. (2011). Putting the pieces together: An integrated model of program implementation. Prevention Science, 12, 23–33. doi:10.1007/s11121-010-0186-1.CrossRefPubMedPubMedCentral

Berry, V., Axford, N., Blower, S., Taylor, R. S., Edwards, R. T., Tobin, K., … Bywater, T. (2016). The effectiveness and micro-costing analysis of a universal, school-based, social–emotional learning programme in the UK: A cluster-randomised controlled trial. School Mental Health, 238–256. doi:10.1007/s12310-015-9160-1

Bierman, K. L., Nix, R. L., Heinrichs, B. S., Domitrovich, C. E., Gest, S. D., Welsh, J. A., & Gill, S. (2014). Effects of head start REDI on children’s outcomes 1 year later in different kindergarten contexts. Child Development, 85, 140–159. doi:10.1111/cdev.12117.CrossRefPubMed

Bruhn, A. L., Hirsch, S. E., & Lloyd, J. W. (2015). Treatment integrity in school-wide programs: A review of the literature (1993–2012). The Journal of Primary Prevention, 36, 335–349. doi:10.1007/s10935-015-0400-9.CrossRefPubMed

Carpenter, J. J. R., Goldstein, H., & Kenward, M. G. M. (2011). REALCOM-IMPUTE software for multilevel multiple imputation with mixed response types. Journal of Statistical Software, 45, 1–12. doi:10.18637/jss.v045.i05.CrossRef

Carroll, C., Patterson, M., Wood, S., Booth, A., Rick, J., & Balain, S. (2007). A conceptual framework for implementation fidelity. Implementation Science, 2, 40–49. doi:10.1186/1748-5908-2-40.CrossRefPubMedPubMedCentral

Conduct Problems Prevention Research Group. (1999). Initial impact of the Fast track prevention trial for conduct problems: II. Classroom effects. Journal of Consulting and Clinical Psychology, 67, 648–657.CrossRefPubMedCentral

Crean, H. F., & Johnson, D. B. (2013). Promoting alternative thinking strategies (PATHS) and elementary school aged children’s aggression: Results from a cluster randomized trial. American Journal of Community Psychology, 52, 56–72. doi:10.1007/s10464-013-9576-4.CrossRefPubMed

Cross, W., West, J., Wyman, P. A., et al. (2015). Observational measures of implementer fidelity for a school-based preventive intervention: Development, reliability and validity. Prevention Science, 16, 122–132. doi:10.1007/s11121-014-0488-9.CrossRefPubMedPubMedCentral

Department for Education. (2012). Schools, pupils and their characteristics. London: Department for Education.

DiStefano, C., Zhu, M., & Mindrila, D. (2009). Understanding and using factor scores: Considerations for the applied researcher. Practical Assessment, Research and Evaluation, 14, 1–11.

Domitrovich, C. E., Cortes, R. C., & Greenberg, M. T. (2007). Improving young children’s social and emotional competence: A randomized trial of the preschool “PATHS” curriculum. The Journal of Primary Prevention, 28, 67–91. doi:10.1007/s10935-007-0081-0.CrossRefPubMed

Domitrovich, C. E., Gest, S. D., Jones, D., Gill, S., & Sanford Derousie, R. M. (2010). Implementation quality: Lessons learned in the context of the head start REDI trial. Early Childhood Research Quarterly, 25, 284–298. doi:10.1016/j.ecresq.2010.04.001.CrossRefPubMedPubMedCentral

Durlak, J. (2010). The importance of doing well in whatever you do: A commentary on the special edition, “implementation research in early childhood education”. Early Childhood Research Quarterly, 25, 348–357. doi:10.1016/j.ecresq.2010.03.003.CrossRef

Durlak, J. A. (2015). Studying program implementation is not easy but it is essential. Prevention Science, 16, 1123–1127. doi:10.1007/s11121-015-0606-3.CrossRefPubMed

Durlak, J. A. (2016). Programme implementation in social and emotional learning: Basic issues and research findings. Cambridge Journal of Education, 46, 333–345. doi:10.1080/0305764X.2016.1142504.CrossRef

Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350. doi:10.1007/s10464-008-9165-0.CrossRefPubMed

Durlak, J. a., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., & Schellinger, K. B. (2011). The impact of enhancing students’ social and emotional learning: A meta-analysis of school-based universal interventions. Child Development, 82, 405–432. doi:10.1111/j.1467-8624.2010.01564.x.CrossRefPubMed

Elliott, D., & Mihalic, S. (2004). Issues in disseminating and replicating effective prevention programs. Prevention Science, 5, 47–53. doi:10.1023/B:PREV.0000013981.28071.52.CrossRefPubMed

EPISCenter. (2014). Promoting alternative thinking strategies. Retrieved August 14, 2014, from http://www.episcenter.psu.edu/ebp/altthinking

Faria, A. M., Kendziora, K., Brown, L., O’Brien, B., & Osher, D. (2013). PATHS implementation and outcome study in the Cleveland metropolitan School District: Final report. Washington: American Institutes for Research.

Goodman, R. (2001). Psychometric properties of the strengths and difficulties questionnaire. Journal of the American Academy of Child and Adolescent Psychiatry, 40, 1337–1345. doi:10.1097/00004583-200111000-00015.CrossRefPubMed

Goodman, A., Lamping, D. L., & Ploubidis, G. B. (2010). When to use broader internalising and externalising subscales instead of the hypothesised five subscales on the strengths and difficulties questionnaire (SDQ): Data from British parents, teachers and children. Journal of Abnormal Child Psychology, 38, 1179–1191. doi:10.1007/s10802-010-9434-x.CrossRefPubMed

Green, S. B. (1991). How many subjects does it take to do a regression analysis. Multivariate Behavioral Research, 26, 499–510. doi:10.1207/s15327906mbr2603_7.CrossRefPubMed

Green, H., McGinnity, A., Meltzer, H., Ford, T., & Goodman, R. (2005). Mental health of children and young people in Great Britain. Newport: Office for National Statistics.

Gresham, F. M., & Elliot, S. N. (2008). Social skills improvement system: Rating scales manual. Minneapolis, MN: Pearson

Hallgren, K. A. (2012). Computing inter-rater reliability for observational data: An overview and tutorial. Tutorials in Quantitative Methods for Psychology, 8, 23–34.CrossRefPubMedPubMedCentral

Hansen, W. (2014). Measuring fidelity. In Z. Sloboda & H. Petras (Eds.), Defining prevention science (pp. 335–359). New York: Springer.CrossRef

Hansen, W. B., Pankrantz, M. M., Dusenbury, L., Giles, S. M., Bishop, D., Albritton, J., et al. (2013). Styles of adaptation: The impact of frequency and valence of adaptation on preventing substance abuse. Health Education, 113, 345–363.CrossRef

Hansen, W. B., Pankratz, M. M., & Bishop, D. C. (2014). Differences in observers’ and teachers’ fidelity assessments. Journal of Primary Prevention, 35, 297–308. doi:10.1007/s10935-014-0351-6.CrossRefPubMed

Heijtan, D. F., & Basu, S. (1996). Distinguishing “missing at random” and “missing completely at random”. The American Statistician, 50, 207–213.

Humphrey, N., Kalambouka, A., Wigelsworth, M., Lendrum, A., Deighton, J., & Wolpert, M. (2011). Measures of social and emotional skills for children and young people: A systematic review. Educational and Psychological Measurement, 71, 617–637. doi:10.1177/0013164410382896.CrossRef

Humphrey, N., Barlow, A., Wigelsworth, M., Lendrum, A., Pert, K., Joyce, C., et al. (2015). Promoting alternative thinking strategies (PATHS): Evaluation report. London: Education Endowment Foundation.

Humphrey, N., Barlow, A., Wigelsworth, M., Lendrum, A., Pert, K., Joyce, C., et al. (2016a). A cluster randomized controlled trial of the promoting alternative thinking strategies (PATHS) curriculum. Journal of School Psychology, 58, 73–89. doi:10.1016/j.jsp.2016.07.002.CrossRefPubMedPubMedCentral

Humphrey, N., Lendrum, A., Ashworth, E., Frearson, K., Buck, R., & Kerr, K. (2016b). Implementation and process evaluation (IPE) for interventions in educational settings: A synthesis of the literature. London: Education Endowment Foundation.

Kam, C.M., Greenberg, M. T., & Walls, C. T. (2003). Examining the role of implementation quality in school-based prevention using the PATHS curriculum. Prevention Science, 4, 55–63–63. doi:10.1023/A:1021786811186

Lendrum, A., Humphrey, N., & Greenberg, M. T. (2016). Implementing for success in school-based mental health promotion: The role of quality in resolving the tension between fidelity and adaptation. In R. Shute & P. Slee (Eds.), Mental health and wellbeing through schools: The way forward (pp. 53–63). London: Taylor and Francis.

Liu, J. (2010). Minimum effective dose. In Encyclopedia of Biopharmaceutical Statistics (pp. 799–800). London: Informa.

Low, S., Smolkowski, K., & Cook, C. (2016). What constitutes high-quality implementation of SEL programs? A latent class analysis of second step implementation. Prevention Science, 17, 981–991. doi:10.1007/s11121-016-0670-3.CrossRefPubMed

Moore, J. E., Bumbarger, B. K., & Cooper, B. R. (2013). Examining adaptations of evidence-based programs in natural contexts. The Journal of Primary Prevention, 34, 147–161. doi:10.1007/s10935-013-0303-6.CrossRefPubMed

Myung, I. J. (2000). The importance of complexity in model selection. Journal of Mathematical Psychology, 44, 190–204. doi:10.1006/jmps.1999.1283.CrossRefPubMed

Nelson, M. C., Cordray, D. S., Hulleman, C. S., Darrow, C. L., & Sommer, E. C. (2012). A procedure for assessing intervention fidelity in experiments testing educational and behavioral interventions. Journal of Behavioral Health Services and Research, 39, 374–396. doi:10.1007/s11414-012-9295-x.CrossRefPubMed

O’Donnell, C. L. (2008). Defining, conceptualising, and measuring fidelity of implementation and its relationship to outcomes in K-12 curriculum intervention research. Review of Educational Research, 78, 33–84.CrossRef

Odom, S. L., Fleming, K., Diamond, K., Lieber, J., Hanson, M., Butera, G., et al. (2010). Examining different forms of implementation in early childhood curriculum research. Early Childhood Research Quarterly, 25, 314–328. doi:10.1016/j.ecresq.2010.03.001.CrossRefPubMedPubMedCentral

Ogden, T., & Fixsen, D. L. (2014). Implementation science: A brief overview and a look ahead. Zeitschrift für Psychologie, 222, 4–11.CrossRef

Pettigrew, J., Graham, J. W., Miller-Day, M., Hecht, M. L., Krieger, J. L., & Shin, Y. J. (2015). Adherence and delivery: Implementation quality and program outcomes for the seventh-grade keepin’ it REAL program. Prevention Science : The Official Journal of the Society for Prevention Research, 16, 90–99. doi:10.1007/s11121-014-0459-1.CrossRef

Schoenwald, S. K., & Garland, A. F. (2014). A review of treatment adherence measurement methods. Psychological Assessment, 25, 146–156. doi:10.1037/a0029715.A.CrossRef

Schonfeld, D. J., Adams, R. E., Fredstrom, B. K., Weissberg, R. P., Gilman, R., Voyce, C., et al. (2015). Cluster-randomized trial demonstrating impact on academic achievement of elementary social-emotional learning. School Psychology Quarterly, 30, 406–420. doi:10.1037/spq0000099.CrossRefPubMed

Sklad, M., Diekstra, R., De Ritter, M., Ben, J., & Gravesteijn, C. (2012). Effectiveness of school-based universal social, emotional, and behavioral programs: Do they enhance students’ development in the area of skills, behavior and adjustment? Psychology in the Schools, 49, 892–909. doi:10.1002/pits.CrossRef

Snijders, T. A. B. (2005). Power and sample size in multilevel modeling. In B. S. Everitt & D. C. Howell (Eds.), Encyclopedia of statistics in behavioral science (pp. 1570–1573). Chichester: Wiley.

Snijders, T. A. B., & Bosker, R. J. (2012). Multilevel analysis. London: SAGE Publications.

Social and Character Development Research Consortium. (2010). Efficacy of school-wide programs to promote social and character development and reduce problem behavior in elementary school children. Washington: Institute of Educational Science.

Weissberg, R. P., Durlak, J. A., Domitrovich, C. E., & Gullotta, T. P. (2015). Social and emotional learning: Past, present and future. In J. A. Durlak, C. E. Domitrovich, R. P. Weissberg, & T. P. Gullotta (Eds.), Handbook of social and emotional learning (pp. 3–19). New York: Guilford Press.

Wigelsworth, M., Lendrum, A., Oldfield, J., Scott, A., Ten-Bokkel, I., Tate, K., & Emery, C. (2016). The influence of trial stage, developer involvement and international transferability on the outcomes of universal social and emotional learning outcomes: A meta-analysis. Cambridge Journal of Education, 46, 347–376. doi:10.1080/0305764X.2016.1195791.

Title: Quality Matters: Implementation Moderates Student Outcomes in the PATHS Curriculum
Authors: Neil Humphrey
Alexandra Barlow
Ann Lendrum
Publication date: 01-02-2018
Publisher: Springer US
Published in: Prevention Science / Issue 2/2018
Print ISSN: 1389-4986
Electronic ISSN: 1573-6695
DOI: https://doi.org/10.1007/s11121-017-0802-4

At a glance: The STEP trials

Springer Medicine

Quality Matters: Implementation Moderates Student Outcomes in the PATHS Curriculum

Abstract

At a glance: The STEP trials

Springer Medicine

Abstract

Please log in to get access to this content

Other articles of this Issue 2/2018

Policy and Prevention Approaches for Disordered and Hazardous Gaming and Internet Use: an International Perspective

Preventing Childhood Anxiety Disorders: Is an Applied Game as Effective as a Cognitive Behavioral Therapy-Based Program?

A Brief Smoking Cessation Advice by Youth Counselors for the Smokers in the Hong Kong Quit to Win Contest 2010: a Cluster Randomized Controlled Trial

Exposure to and Content of Marijuana Product Reviews

Examining the Highs and Lows of the Collaborative Relationship Between Technical Assistance Providers and Prevention Implementers

Testing the Amotivational Syndrome: Marijuana Use Longitudinally Predicts Lower Self-Efficacy Even After Controlling for Demographics, Personality, and Alcohol and Cigarette Use