An invitation to Measure the Impact of Professional Learning

In 2004, I became one of the first P12 online professional learning developers in Colorado. I was newly graduated with my PhD in Instructional Design, a branch of educational technology that studies adult learning strategies within the design and facilitation of training. Training is defined broadly as professional learning, in-person workshops, remote synchronous sessions, games, informal learning opportunities, on-the-job experiences, social learning, self-paced asynchronous sessions, communities of practice, professional learning communities, MOOCs, coaching and mentoring (Biech, 2017).

My job was to manage, design, implement and market online P-12 PD to district leaders across Colorado to support teachers’ development “just-in-time” with the convenience of “any-time, anywhere.” Seventeen years ago, leaders were skeptical about the credibility of online PD and could not fathom virtual PD ever replacing face-to-face sessions. My goal was to show how professional learning can change teachers’ practices and impact learning.  

My first online course focused on strategies for teaching English language learners (ELL). I consulted with and filmed district ELL coordinators demonstrating best practices within classrooms. Then, I designed knowledge-building lessons that included the short videos of selected research-based best practices. Teachers then practiced the strategies within their own classroom and posted reflections on the experience within the discussion board. In some districts, instructional coaches would also observe these practices in the classroom and provide feedback. This approach of demonstration, practice, and reflection provided evidence of shifts happening in teachers’ practices (Summers & Selzer, 2007). Eventually, hand-held video cameras, iPads, or Swivels also allowed teachers to record themselves as evidence of practice which extended opportunities for virtual coaching.

While the need for virtual PD is more evident than ever before, its effectiveness continues to be scrutinized. There are ways to measure the impact of professional learning and Guskey offers a model that has been tested for over twenty years.  

Similar to the process of backwards planning, Guskey’s five-level model asks educators to begin at Level 5: Improved Learner Outcomes by asking the question, “What outcomes do you want to achieve and what sources are data will determine the results of those goals?” (Guskey, Roy, & von Frank, 2014). The table below illustrates each level, what is being measured, and what questions to ask at each level.

Evaluation Level	What is Measured? Questions to ask:
Level 1: PD Quality/Participants’ Reactions	Initial satisfaction with the experience:  Is the PD well-designed pedagogically? Did the participants like it? Is it accessible to its intended audience? Do intended outcomes align with our system’s priorities and what our educators need?
Level 2: Participants’ Learning	New knowledge and skills of the participants:  Did participants acquire the intended knowledge and skills?   Was the content engaging?
Level 3: Organizational Support & Change	The organization’s advocacy, support, accommodation, facilitation, and recognition:  What was the impact on the organization? Were successes recognized and shared? Did the PD affect the climate, culture and processes? Did a teacher’s attitude change a result of the PD?
Level 4: Change in Practice/Use of New Knowledge and Skills	Did the PD create a sustained change in teaching practice? What PD efforts have the greatest success in changing practice and what elements do they have in common?
Level 5: Change in Learner Outcomes	Did the change in practice impact student outcomes? Did the changes drive improvements in some context (e.g., schools/subjects and not others)? What other factors could have impacted the outcomes?

Adapted from Guskey, T. (2002). Does it make a difference? Evaluating professional development. Educational Leadership 59(6), 45-51 and Learning Forward (2016). Guskey levels for the pd redesign community.

I have seen results using this evaluation model within a large urban school district. Now, I invite you to try this model and to reach out if you have questions (laura.summers@ucdenver.edu).  

 [This piece was originally published as my contribution to a weekly Monday Morning Message for the School of Education and Human Development doctoral studies community at the University of Colorado Denver on March 15, 2021.]

References:

Biech, E. (2017). The art and science of training. ATD.

Guskey, T. (2002). Does it make a difference? Evaluating professional development. Educational Leadership 59(6), 45-51.

Guskey, T., Roy, P., & van Frank, V. (2014). Research the highest standard in professional learning: Data. Learning Forward and Corwin.

Learning Forward (2016). Guskey levels for the pd redesign community. Learning Forward.

Summers, L. & Selzer, D. (2007). Content templates. In P. Shank’s (Ed.) The Online Learning Idea Book. San Francisco:  Pfeiffer.