Overview
Evaluation of student performance and program effectiveness is an important step in the teaching and learning cycle. Evaluation allows educators to assess program effectiveness, identify areas for improvement, and if using pre-curriculum assessments, gauge learners’ prior knowledge. Evaluative procedures must be informed by stated goals and objectives, emphasizing the key points learners should take from your teaching session.
Evaluation can take many forms, from simple satisfaction surveys to more intense direct observation and feedback. The methods of assessment should be tailored to your desired outcomes. Here we will present the Kirkpatrick Evaluation Model as a way to conceptualize assessment practices.
“There are no mistakes or failures, only lessons.” - Denis Waitley
Key Learning Points
- Evaluation practices must be aligned with stated goals and objectives.
- The Kirkpatrick Evaluation Model serves as a guide to assessing students and programs on multiple levels, with level one representing learner reactions and level four the impact of student learning on target outcomes.
- Results of evaluations should be used to inform revisions to your teaching sessions.
Guiding Principles
Kirkpatrick’s Evaluation Model
The Kirkpatrick Evaluation Model (left) is a widely used paradigm to discuss evaluation of teaching effectiveness. The impact of a program can be assessed on four levels, with level one, reaction, reflecting learner satisfaction or dissatisfaction, to level four, which assesses outcomes based on goals and objectives set forth by the program. Throughout this module, we will use an example of a teaching session on managing sepsis to illustrate these principles.
Level One: Reaction
Level one evaluation assesses how learners felt about the teaching session. For example, did they enjoy the session? Were they satisfied with the content? Do they believe the content is relevant?
Often, this feedback is gathered with immediate post-session evaluation forms and is relatively easy to analyze for future improvements. Reactionary feedback does not assess participant learning or behavioral change.
Example: For a teaching session on managing sepsis, you might ask the following of participants:
- On a scale of one (did not enjoy at all) to five (enjoyed very much), indicate how much you enjoyed this session on managing sepsis.
- How relevant do you believe this session on managing sepsis is for your internship preparation?
Level Two: Learning
Level two evaluation practices assess how student knowledge has changed after a program. That is, did they learn what they were supposed to learn?
Learning evaluation is often achieved through pre- and post-program assessments of knowledge. This can take the form of self-assessment or written or verbal quizzes or tests, for example. This provides more meaningful data than reactionary feedback, however, it can be difficult to develop a measure for abstract skills and can be costly to implement.
Example: For a teaching session on managing sepsis, you could assess individual learning with a brief written quiz. For group assessment, you could use an audience response system to test knowledge-based questions.
Level Three: Behavior
Level three evaluations assess the degree to which learners applied the knowledge gained from the session to change their behavior. For example, did students apply the concepts discussed in class to a real-world scenario?
Evaluation for behavior change often requires observation of a learner in their environment to determine whether they are employing skills and knowledge taught in the session. This is highly meaningful assessment data, but is challenging to implement in the context of a single teaching session.
Example: To determine whether students can apply knowledge gained in a teaching session on sepsis, one could observe a student’s approach to managing sepsis in a real-world scenario. For example, when presented with a septic patient, did they manage fluid resuscitation, antimicrobials, and source control appropriately? In the absence of real-world assessment, simulation can be used as a proxy.
Level Four: Results
Evaluation of results assesses the impact of the enhanced learner skill on their environment. For example, how are patient outcomes affected by learner skill development? How are team dynamics or patient satisfaction impacted by communication skill development?
This is the highest level of evaluation and is the most difficult to implement, particularly on a large scale. With the lack of longitudinal contact with your learners in the Internship Preparation Course, this level of evaluation is unlikely to be achieved.
Example: If students are being followed longitudinally, one could assess the outcomes of the septic patients managed by students participating in the teaching session on sepsis. Does their knowledge of sepsis management translate to improved patient outcomes?
Wrap Up
Well-planned evaluation procedures closely aligned with session goals and objectives are necessary to effectively assess student learning and to inform subsequent revisions of your teaching session. The Kirkpatrick Model can be used to plan assessment procedures to evaluate student learning on multiple levels.
Other Resources
Gross Davis, B. Tools for Teaching. Second Ed. Jossey-Bass, 2009.
Lang, J.M. On Course: A Week-by-Week Guide to your First Semester of College Teaching. Harvard University Press, 2008.
Schuh, J.H., Biddix, J.P., Dean, L.A., and Kinzie, J. Assessment in Student Affairs. Second Ed. Jossey-Bass, 2016.
Skeff, K.M., and Stratos, G.A. Methods for Teaching Medicine. American College of Physicians Press, 2010.
References
Bree, K.K., Whicker, S.A., Fromme, H.B., Paik, S., and Greenberg, L. "Resident-as-teachers publications: What can programs learn from the literature when starting a new or refining an established curriculum?" Journal of Graduate Medical Education. 2014. 237-48.
Kirkpatrick Partners, LLC. The Kirkpatrick Model. 2017. Retrieved from https://www.kirkpatrickpartners.com/Our-Philosophy/The-Kirkpatrick-Model
Suski, L. Assessment Student Learning: A Common Sense Guide. Second Ed. Jossey-Bass, 2009.