Differences in Observers’ and Teachers’ Fidelity Assessments

As evidence-based programs become disseminated, understanding the degree to which they are implemented with fidelity is crucial. This study tested the validity of fidelity ratings made by observers versus those made by teachers. We hypothesized that teachers’ reports about fidelity would have a positivity bias when compared to observers’ reports. Further, we hypothesized that there would generally be low correspondence between teachers’ and observers’ ratings of fidelity. We examined teachers’ and observers’ ratings as they were related to mediating variables targeted for change by the intervention. Finally, we examined the role that years of teaching experience played in achieving fidelity. Eighteen teachers and four research assistants participated in this project as raters. Teachers made video recordings of their implementation of All Stars and completed fidelity assessment forms. Trained observers independently completed parallel forms for 215 sampled classroom sessions. Both teachers and observers rated adherence, quality of delivery, attendance, and participant engagement. Teachers made more positive fidelity ratings than did observers. With the exception of ratings for attendance, teachers and observers failed to agree on fidelity ratings. Observers’ ratings were significantly related to students’ pretest assessments of targeted program mediators. That observers’ ratings were related to students’ pretest scores, suggests it is easier to teach well when students are predisposed to program success. Teachers’ ratings were infrequently related to mediators, but when they were, the relationship was counterintuitive. Experienced teachers taught with greater fidelity than novice teachers. While possibly inflated and inaccurate, gathering fidelity assessments from teachers may sensitize them to issues of fidelity as a result of requiring form completion. Assessing fidelity through observers’ ratings of video recordings has significant merit. As a longterm investment in improving prevention outcomes, policy makers should consider requiring both teacher and observer fidelity assessments as essential components of evaluation.