PHE 423 Module 5 Project One Example

Will the Alarms Still Sound? An Evaluation Plan for the River City Smoke Alarm Program

[Student Name]

Southern New Hampshire University

PHE 423: Evaluation Methods in Public Health

Project One

[Instructor Name]

[Date]

The organization, setting and figures below are a composite written as a model document. No real employer, client, colleague or patient is described.

Will the Alarms Still Sound? An Evaluation Plan for the River City Smoke Alarm Program

After three people died in house fires one winter in homes with no working alarm, the fire department and the county health department of a river city in southeast Missouri began putting long-life sealed alarms, at no cost, into rental houses built before 1970. Crews go door to door in two neighborhoods this year; a third, similar in housing age and income, is scheduled for next year. This plan describes how the program will be evaluated over its first twelve months. It follows the six steps of the CDC framework, from engaging stakeholders to ensuring use (Centers for Disease Control and Prevention, 1999), and draws on the program logic model built in Module Three.

Stakeholders and What They Need

Four groups have a stake in the findings. The fire chief wants to know whether crews' time is well spent. The health department director must report to the county commission, which funds half the alarms. Landlords control access to the homes and worry about liability. Residents, many of them older or renting month to month, want safety without intrusion. A five-person advisory group, with a landlord and two residents among its members, will review the questions, the survey and the draft report.

Evaluation Questions

The plan asks four questions, two about delivery and two about results. First, what share of eligible homes in the program neighborhoods received alarms, and which homes were missed? Second, were alarms installed as intended, with one on every sleeping level and a short safety talk? Third, twelve months after installation, what share of served homes have working alarms, compared with the neighborhood not yet served? Fourth, what share of households report a home escape plan? Fire injuries and deaths are tracked but are not treated as a test of the program, because in a city this size they are too rare to show change in one year.

Two further questions are reserved for later years if funding continues: whether alarms are still working at three years, when the risk of removal grows, and whether the cost per working alarm falls as crews gain experience. Keeping the first year to four questions protects the quality of the data on each.

Design

The evaluation uses a staggered comparison. The two program neighborhoods are compared with the third, whose turn comes next year, so no household is held back for the sake of the study. Randomizing homes was considered and dropped after the fire chief judged it unfair. The weakness of this design is that the neighborhoods are not identical. To reduce that risk, the same door-to-door survey of working alarms will be completed in random samples of homes in all three neighborhoods before installation begins, and results at twelve months will be reported as change from that starting point.

Indicators and Data Sources

Each question has one main indicator and a source the program already collects or can collect cheaply. The table lists them.

Table 1. Indicators and sources

Note. Every indicator is tied to a question the stakeholders asked for.

Sampling and Data Collection

Crews will log every home they knock on, whether anyone answered, whether the household agreed and how many alarms were installed. At twelve months, a random sample of 200 served homes and 150 homes in the comparison neighborhood will be revisited by trained health department staff rather than the installers, so the people who put alarms up are not judging them. With permission, the visitor will press each alarm's test button and ask the five survey items. Up to three visits at different times of day will be made before a home is counted as unreached.

Visitors will carry a short script, a photo badge and a letter from the fire chief, because many older residents are wary of strangers at the door. If an alarm fails the test, the visitor will replace it on the spot and record the reason, such as a removed unit or a missing alarm, which feeds the analysis of why alarms stop working.

Analysis

Reach and fidelity will be reported as simple percentages with the reasons homes were missed. For working alarms, the share in served homes will be compared with the comparison neighborhood, with 95% confidence intervals, and change from baseline will be shown for both. A difference of ten points or more would be treated as meaningful for the program. DiGuiseppi et al. (2002) found that giving out alarms in London did not reduce injuries, partly because many were never fitted or were later disabled, so the plan places working alarms at twelve months at the center of the analysis. Yellman et al. (2018) estimated large net benefits for an installation program in Dallas, which gives the city a benchmark for any later economic analysis.

Ethics and Burden

Households may refuse any part of the revisit, and refusing will never affect whether they keep their alarms. No names will appear in reports; addresses will be stored apart from survey answers. Landlords will be told that the evaluation is about the program, not inspection of their property. The survey takes under five minutes.

Timeline and Budget

Baseline surveys run in months one and two, installation in months two through eight, revisits in months fourteen through sixteen and the report in month eighteen. The budget is modest because the program already collects most of the data.

Table 2. Evaluation budget

Note. Totals sum to $15,000, about 8% of program cost.

Use of Findings

Glasgow et al. (1999) held that how much a program improves population health rests on reach and maintenance as much as effect. The plan therefore reports reach and working alarms side by side, so leaders can see whether the next neighborhood needs a different way to reach renters. Following the Kellogg guide's advice to share results with those who shaped the program (W. K. Kellogg Foundation, 2004), the advisory group will see findings first, and a one-page summary will go to landlords and residents.

References

Centers for Disease Control and Prevention. (1999). Framework for program evaluation in public health. MMWR Recommendations and Reports, 48(RR-11), 1-40.

DiGuiseppi, C., Roberts, I., Wade, A., Sculpher, M., Edwards, P., Godward, C., Pan, H., & Slater, S. (2002). Incidence of fires and related injuries after giving out free smoke alarms: Cluster randomised controlled trial. BMJ, 325(7371), 995-998. https://doi.org/10.1136/bmj.325.7371.995

Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health, 89(9), 1322-1327. https://doi.org/10.2105/AJPH.89.9.1322

W. K. Kellogg Foundation. (2004). Logic model development guide. W. K. Kellogg Foundation.

Yellman, M. A., Peterson, C., McCoy, M. A., Stephens-Stidham, S., Caton, E., Barnard, J. J., Padgett, T. O., Florence, C., & Istre, G. R. (2018). Preventing deaths and injuries from house fires: A cost-benefit analysis of a community-based smoke alarm installation programme. Injury Prevention, 24(1), 12-18. https://doi.org/10.1136/injuryprev-2016-042247