Iowa Measurement and Research Foundation Final Report

Submitted by
Dr. Allison Bruhn, Assistant Professor or Special Education
Department of Teaching and Learning
The University of Iowa


The purpose of Project RATE IT was to learn how to better support students with behavior problems through effective intervention and accurate progress monitoring. Specific research activities included (a) reviewing the literature on behavioral goal-setting, and (b) conducting a concurrent validity study examining the relation between teacher ratings and outside observer ratings of behavior, as well as these ratings’ relation to direct observation of academic engagement.

Summary of Work

First, we conducted a systematic literature review of behavioral goal-setting studies conducted with K-12 students who had or were at risk for emotional/behavioral disorders (EBD). The purpose of this review was to gain a better understanding of typical behavioral goals for students with EBD and how these goals are incorporated into interventions and then how progress toward goals is tracked. The review included 40 studies on behavioral goal-setting, which were analyzed and discussed in terms of participants, setting, design, measures, intervention components, and student outcomes.  The most important findings from this review included (a) the lack of student input into behavioral goal setting interventions, (b) the frequent use of data for monitoring progress and providing feedback.  This systematic review resulted in two manuscripts (one in press, one in review) and a conference presentation:

Bruhn, A. L., McDaniel, S. M., Fernando, J, & Troughton, L. (in press). Goal-setting for students with significant behavior problems: A systematic review of the literature. Behavioral Disorders.

Bruhn, A. L., Fernando, J., McDaniel, S. M., & Troughton, L. (in review, 2016). Putting behavioral goal-setting research into practice. Manuscript submitted for publication at Beyond Behavior.

Bruhn, A. L., Troughton, L., & McDaniel, S. (2015, September). Incorporating goal-setting into behavioral interventions: What does the research say? A presentation at the Council for Children with Behavior Disorders. Atlanta, GA.

After finishing the systematic review indicating that teachers need to use data to individualize goals and monitor students’ behavioral progress, we conducted a validity study to examine potential behavioral progress monitoring tools. Across three schools, 12 consented 5th-6th grade teachers nominated two students each—one with positive classroom behavior and one with negative classroom behavior—resulting in 24 student participants.  Each participant was rated on a behavioral rating scale called the Strengths and Difficulties Questionnaire (SDQ; Goodman, 1997) at the beginning of the study.  Then, each student was observed on eight separate occasions during 20 minutes of whole-group reading/language arts instruction. During these sessions, research assistants collected data on the two consented students’ academic engagement. After the session, the research assistants and teachers independently rated the students on their adherence to schoolwide behavioral expectations (i.e., Be Respectful, Be Responsible, Be Ready) using a 0-4 scale (0 = never, 0%, not at all; 1= a little, 25%, rarely; 2 = sometimes, 50%, about half the time; 3 = a lot, 75%, most of the time; 4 = always, 100%, all the time).  This resulted in 190 observations across 24 students.

In terms of data analysis, there were three sets of correlations of interest: 1) correlations between research assistant ratings of behavior and direct observations of engagement, 2) correlations between teacher ratings of behavior and direct observations of engagement, and 3) correlations between teacher ratings of behavior and research assistant ratings of behavior. Results indicated high correlations between researcher ratings and engagement, moderate to high correlations between teacher ratings and engagement, and moderate to high correlations between researcher and teacher ratings of behavior.  Although these findings lend evidence to the validity of teacher ratings as a potential alternative to directly observed academic engagement, significant differences were found between teacher and researcher ratings indicating teachers tended to rate students with behavior problems higher than researchers did.

This study has resulted in one manuscript (in review) and one presentation:

Bruhn, A. L., Barron, S., Fernando, J. A., & Balint-Langel, K. (in review, 2016).  Extending the DBR: A comparison of schoolwide behavior ratings and academic engagement. Manuscript submitted for publication at Education and Treatment of Children.

Bruhn, A. L. (2015, October). Show me the data!. A presentation at the Teacher Educators for Children with Behavioral Disorders 39th Annual Conference. Tempe, AZ.

Next Steps

Next steps include examining the data from the validity study within the context of the systematic review on goal-setting. That is, can we determine a reasonable behavioral goal for teacher ratings of behavior and/or academic engagement? An additional step includes examining data sets to establish a set of progress-monitoring decision rules that can be used when behavior ratings serve as the measurement tool for behavioral progress during intervention.