NUR 634 Module 7 Milestone Three Example

Milestone Three: A Four-Level Evaluation Plan for the Flipped Heart Failure Session

[Student Name]

Southern New Hampshire University

NUR 634: Facilitating Learning and Teaching Innovation in Nursing Education

Milestone Three

[Instructor Name]

[Date]

The organization, setting and figures below are a composite written as a model document. No real employer, client, colleague or patient is described.

Milestone Three: A Four-Level Evaluation Plan for the Flipped Heart Failure Session

Teaching innovations are often judged by how students feel about them. A session students enjoy may not improve learning, and a session they find difficult may improve it considerably. An evaluation plan needs to measure more than satisfaction and to be honest about what its design can show. This milestone plans the evaluation of the flipped heart failure session at Riverbend Community College. It argues that measuring reaction, learning, clinical behavior and results, comparing with earlier cohorts and naming the threats to validity will give the curriculum committee a sound basis for deciding the innovation's future.

Choosing an Evaluation Model

A guide to program evaluation in health professions education describes several models, including experimental and quasi-experimental designs, the Kirkpatrick levels, logic models and the CIPP model of context, input, process and product, and it emphasizes choosing a model that fits the questions stakeholders need answered and the resources available (Frye & Hemmer, 2012). The curriculum committee's questions are practical: did students learn more, did they apply it in clinical and was the effort worthwhile? Kirkpatrick's four levels fit those questions well. They move from reaction, how participants respond, to learning, what knowledge or skill they gain, to behavior, whether they apply it, to results, the broader outcomes that follow (Kirkpatrick & Kirkpatrick, 2006). The model is widely used and easy to explain, although critics note that it assumes each level causes the next, which is not always true.

Level 1: Reaction

Immediately after the session, students will complete a six-item survey on whether the pre-class materials were clear and manageable in length, whether the case helped them understand heart failure, how confident they feel caring for a patient with heart failure and what should change. Two open questions will capture comments. Reaction data matter because the nursing review of flipped classrooms found that satisfaction was mixed and that engagement improved when students understood the purpose of the model; low reaction scores could signal a problem with explanation or workload that the design can fix. Reaction alone will not be treated as evidence of learning.

Level 2: Learning

Learning will be measured with the heart failure unit exam, which has been used with only minor changes for three years. The primary comparison is the average score on the eight application items, which averaged 61% across the previous three cohorts, against the pilot cohort, with a target of at least 70%. Item analysis will check the quality of those items. The difficulty index, the proportion of students answering correctly, should fall between roughly 30% and 80% for useful items. The discrimination index, which compares how well high-scoring and low-scoring students do on an item, should be positive and preferably above 0.2; an item that low scorers answer correctly as often as high scorers tells faculty little. Items with poor discrimination will be reviewed before results are interpreted, so that a weak item does not mask real learning or create a false gain.

Level 3: Behavior

The innovation's purpose is to change what students do in clinical, not only on exams. Clinical faculty will rate each student with the Lasater clinical judgment rubric on the clinical day following the session, focusing on focused observation, recognizing deviations from expected patterns and prioritizing data, the dimensions most related to the session's objectives. Because the rubric grew out of Tanner's model, its four-step scale gives faculty a common yardstick for these behaviors (Lasater, 2007). Before rating, faculty will rate two recorded student simulations together to calibrate their use of the rubric. The comparison will be with ratings from the previous cohort, collected in the same week with the same rubric, and with the target that at least half of students reach the developing level or above on all three dimensions.

Level 4: Results

Results are the hardest level to attribute to a single session. The plan will track two indicators with appropriate caution: the course pass rate and, later, performance on the heart failure content area of the program's standardized end-of-program examination. Neither can be credited to one session, since many factors affect them, but a consistent pattern across levels would strengthen the case that the innovation contributes. Faculty time will also be recorded, including the 30 hours spent preparing materials, so the committee can weigh benefit against cost. Because materials are reusable, the preparation cost falls sharply in later terms, and the plan will estimate the per-term cost after the first year so that the committee is not comparing a one-time investment against a recurring benefit.

Threats to Validity

Several factors could make the pilot look better or worse than it is. The pilot cohort may differ from earlier cohorts in ability, so admission averages and first-semester grades will be compared. Faculty enthusiasm for a new approach can raise results through extra attention, a form of novelty effect. The exam items were familiar to faculty, who might unintentionally teach to them; to limit this, the case writers did not see the exam items. Students may share information about the case with later sections, though this pilot has one section. Finally, one term is a small sample, so the committee should treat results as a signal for continued testing rather than proof. Where possible, the pilot will be repeated the following term to see whether any gain holds.

Decision Rule

The committee will continue and extend the flipped approach to a second topic if the application item average reaches at least 70%, the rubric target is met and reaction scores do not show serious workload problems. If learning improves but reaction is poor, the design will be revised before extension. If neither learning nor behavior improves, the session will return to its previous format, with the lessons documented.

Conclusion

Evaluating the flipped session at four levels, against a historical baseline, with quality-checked exam items, calibrated clinical ratings and honest attention to threats, will tell Riverbend whether the innovation improves what students know and do. The final project will bring the proposal, design and evaluation together.

References

Frye, A. W., & Hemmer, P. A. (2012). Program evaluation models and related theories: AMEE Guide No. 67. Medical Teacher, 34(5), e288-e299. https://doi.org/10.3109/0142159X.2012.668637

Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating training programs: The four levels (3rd ed.). Berrett-Koehler.

Lasater, K. (2007). Clinical judgment development: Using simulation to create an assessment rubric. Journal of Nursing Education, 46(11), 496-503. https://doi.org/10.3928/01484834-20071101-04