IHP 670 Module 7 Evaluation Design Paper Example

Will We Know If It Worked? An Evaluation Design for Steady Steps

[Student Name]

Southern New Hampshire University

IHP 670: Program Design, Planning and Evaluation

Module Seven Paper

[Instructor Name]

[Date]

The organization, setting and figures below are a composite written as a model document. No real employer, client, colleague or patient is described.

Will We Know If It Worked? An Evaluation Design for Steady Steps

Funders, county commissioners and senior center directors will all ask whether Steady Steps worked, but they mean different things. The funder wants to know whether falls declined; commissioners want to know how many residents it reached; directors want to know whether it was worth hosting. This paper designs an evaluation that answers each question credibly within the program's budget.

A Framework for the Process

Milstein and Wetterhall (2000) described the CDC framework for program evaluation as six linked steps that begin with bringing interested parties to the table and describing the program, move through choosing a focus and collecting credible evidence and end with defensible conclusions and deliberate use of what was learned. Four standards, utility, feasibility, propriety and accuracy, guide choices at each step. The framework keeps evaluation tied to decisions people will make rather than to data collection for its own sake.

Engaging Stakeholders and Focusing Questions

An evaluation workgroup of the funder's program officer, two senior center directors, a primary care physician, the department's epidemiologist and two older adults settled on five evaluation questions. Did the program reach priority groups? Did participants' balance improve and falls decline? Did senior centers and clinics adopt it? Was it delivered as designed? Will it continue after the grant?

Organizing Indicators With RE-AIM

Glasgow et al. (1999) proposed judging health promotion programs on five dimensions. Reach asks how many eligible people join and whether they resemble the population at risk. Effectiveness asks what changes follow, good or bad. Adoption asks which settings take the program up. Implementation asks how faithfully staff deliver it. Maintenance asks whether both the benefits and the delivery persist over time. In their view, weakness on any one dimension shrinks what a program achieves for the population. The five evaluation questions map directly onto these dimensions.

Indicators

The table lists indicators, data sources and targets for each dimension.

Table 1. Evaluation Indicators by RE-AIM Dimension

Note. Composite indicators for the Steady Steps evaluation.

Evaluating a Complex Intervention

Craig et al. (2008) advised that evaluations of complex interventions combine outcome measures with process evaluation that explains how and why results occurred, choose the strongest design that is feasible and pay attention to context. Steady Steps' evaluation follows that advice by pairing outcome data with implementation data and interviews, so that a disappointing result can be traced to either the program or its delivery.

Measuring Participant Outcomes

Participants will complete three standard physical tests at enrollment, at the end of the class series and six months later: a timed test of rising, walking and returning to a chair, a thirty-second chair stand and a four-stage balance test. They will also complete a validated fear-of-falling questionnaire. Falls will be recorded on monthly calendars mailed back for six months before and after classes, with a follow-up call when a calendar is missed.

A Comparison Group

Pre-post comparisons alone cannot separate program effects from natural change. Because demand is expected to exceed class capacity in the first year, people on waiting lists will serve as a comparison group, completing the same measures during their wait before joining classes. This wait-list design is fair, since everyone eventually receives the program, and it gives far better evidence than tracking participants alone.

County-Level Trends

For the long-term outcome, the department's epidemiologist will track monthly emergency visits and admissions for falls among adults 65 and older using hospital discharge data, comparing trends before and after the program with those in two similar neighboring counties without such programs. Because many factors affect county rates, this analysis will be interpreted cautiously.

Handling Missing Data and Dropout

Older adults in community programs miss sessions, move or become ill, so some outcome data will be missing. The plan anticipates follow-up rates of about 75% at six months. To limit loss, participants will receive reminder calls and a small grocery card for returning completed fall calendars. Analyses will compare participants who complete follow-up with those who do not on baseline characteristics, and results will be reported for everyone who enrolled as well as for those who attended most sessions, so that the program is not credited only with its most committed participants.

Process Evaluation

Beyond fidelity scores and attendance, the evaluator will interview instructors, site directors and twenty participants each year to understand what helped and hindered delivery, and will review referral completion rates by clinic. These data will explain variation between sites and guide improvements during the program.

Ethics and Data Handling

Participants will give written consent for assessments and data use. Records will sit on a protected county server, with names kept apart from assessment results. The county's human subjects review board will review the plan. Participants may decline assessments and still attend classes.

Analysis and Reporting

Balance scores will be compared before and after classes and against the wait-list group, and fall rates will be expressed as falls per person per month. Reports will go to partners every six months, with a brief summary for the advisory group and a public annual report. Findings will be discussed at partner meetings to decide what to change, completing the framework's final step.

Budget and Limitations

Evaluation will cost about $90,000 over three years, about 7.5% of the grant, mostly for the analyst's time and assessment training. Limitations include self-reported falls, loss to follow-up and the difficulty of attributing county trends to the program.

Conclusion

By following the CDC framework, organizing indicators with RE-AIM and combining outcomes with process data and a wait-list comparison, the Steady Steps evaluation can tell each stakeholder what they need to know and explain why the program succeeded or fell short.

References

Craig, P., Dieppe, P., Macintyre, S., Michie, S., Nazareth, I., & Petticrew, M. (2008). Developing and evaluating complex interventions: The new Medical Research Council guidance. BMJ, 337, Article a1655. https://doi.org/10.1136/bmj.a1655

Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health, 89(9), 1322-1327. https://doi.org/10.2105/AJPH.89.9.1322

Milstein, B., & Wetterhall, S. (2000). A framework featuring steps and standards for program evaluation. Health Promotion Practice, 1(3), 221-228. https://doi.org/10.1177/152483990000100304