HCM 440 Module 3 Study Design Short Paper Example

Designing an Honest Test: Choosing a Study Design for Fairhaven's Text Reminders

[Student Name]

Southern New Hampshire University

HCM 440: Healthcare Research and Evaluation Methodologies

Module Three Short Paper

[Instructor Name]

[Date]

The organization, setting and figures below are a composite written as a model document. No real employer, client, colleague or patient is described.

Designing an Honest Test: Choosing a Study Design for Fairhaven's Text Reminders

The refined question from Module Two asks whether two-way text reminders reduce no-shows compared with automated phone calls. Answering it means estimating what would have happened in the pilot clinics without the texts, a situation nobody can observe directly. Every study design is a strategy for approximating that missing comparison. This paper describes the threats a design must handle, weighs five options against Fairhaven's circumstances and recommends a combined design.

Threats a Design Must Handle

Shadish et al. (2002) describe the main threats to internal validity, meaning reasons other than the program that could explain a change. History refers to other events occurring at the same time, such as Fairhaven's new Saturday hours. Maturation and seasonal patterns refer to changes that would occur anyway; no-shows at Fairhaven are about two points lower in summer than winter. Selection refers to differences between the groups compared. Regression to the mean occurs when groups are chosen because of an unusually bad period, since extreme results tend to drift back toward normal on their own.

Fairhaven's Constraints

Three facts shape the choice. First, six clinics already use texts, and they were not chosen at random: their managers volunteered, and two of them had just finished a winter with unusually high no-show rates. Second, leadership intends to roll texts out to the other eight clinics within a year regardless of the evaluation. Third, the scheduling system holds reliable monthly appointment data for every clinic going back more than two years. A good design will turn these facts into advantages rather than ignore them.

Option 1: Simple Before and After

The easiest design sets the six pilot clinics' no-show rates from the winter months against their spring and early summer rates, which produced the 18.2% to 14.9% figure. It is cheap and quick but vulnerable to nearly every threat. The Saturday hours began a month after the texts, which is history. The comparison runs from winter into summer, which is seasonality. And the volunteers included clinics coming off a bad winter, which invites regression to the mean. The drop could be real, but this design cannot separate the program from these explanations.

Option 2: Patient-Level Randomized Trial

A randomized trial would assign individual patients by chance to text or phone reminders, balancing known and unknown differences between groups. It offers the strongest causal evidence and is how the trials in Guy and colleagues' review were done. At Fairhaven, though, the six pilot clinics have already switched all patients to texts, so switching some back would confuse staff and patients. Within a clinic, front-desk staff would also treat patients differently depending on how they were reminded, contaminating the comparison. A patient-level trial is ideal in principle but poorly suited here.

Option 3: Stepped Wedge Rollout

Hemming et al. (2015) describe the stepped wedge cluster randomized trial, in which groups such as clinics switch from the old practice to the new one at randomly determined times until all have switched. It suits situations where an organization plans to adopt a program everywhere anyway, because nobody is permanently denied it. For Fairhaven's eight remaining clinics, the order could be randomized into four steps of two clinics each, two months apart. Comparing clinics that have switched with those still waiting, month by month, gives randomized evidence while the rollout proceeds on schedule.

Option 4: Interrupted Time Series

This approach relies on a long run of repeated measurements that straddle the start date. Wagner et al. (2002) explain segmented regression, which estimates both a sudden change in level and a change in slope after the intervention, while accounting for the trend beforehand. Lopez Bernal et al. (2017) add that the analysis should model seasonal patterns and check for other events near the start date. Fairhaven has 24 monthly measurements before March for each clinic, enough to estimate trends and seasonality. The weakness is history: a single series cannot separate texts from Saturday hours that began soon after.

Option 5: Difference-in-Differences

Dimick and Ryan (2014) describe difference-in-differences, which compares the change over time in a group that received a program with the change in a group that did not. If both groups would have followed parallel trends without the program, the difference between their changes estimates its effect. Seasonal patterns that affect all clinics cancel out. Fairhaven's eight phone-reminder clinics provide the comparison group. The key assumption, parallel trends, can be checked by plotting the two groups' no-show rates over the 24 months before March.

Comparing the Options

Table 1 compares the five designs on the threats they control, their main weakness and their feasibility at Fairhaven. No single design is best on every measure. The randomized options are strongest against selection but only the stepped wedge is practical. The time series and difference-in-differences designs make good use of existing data, and a controlled time series combines their strengths.

Table 1. Five Designs Compared

Note. Assessment by the author using Shadish et al. (2002) and the methods sources cited.

The Recommended Design

The evaluation will have two parts. Part one is a controlled interrupted time series for the six pilot clinics, using the eight phone-reminder clinics as the comparison until they begin switching. This combines difference-in-differences logic with the trend and seasonality modeling of segmented regression. Part two is a stepped wedge rollout for the eight remaining clinics, with the switching order chosen by lottery. Randomization in part two protects against the selection problem that weakens part one, and if both parts point the same way, leadership can be confident.

Handling the Saturday Hours

The Saturday change is the clearest rival explanation for part one. Three steps address it. The primary analysis will include only weekday appointments, since Saturday visits did not exist before. A secondary analysis will include all visits with a variable marking when Saturday hours began. And because none of the eight remaining clinics plans to add weekend hours, part two will be free of this problem entirely.

Remaining Threats

Some threats remain. Patients who visit clinics in both groups may receive both types of reminders; the analysis will flag and test excluding them. Staff in pilot clinics may work harder to fill slots because they know they are being watched. The vendor may change message wording mid-study, so any change will be logged. And results from one medical group may not apply elsewhere. Skivington et al. (2021) note that evaluations of complex interventions should examine context and implementation as well as outcomes, which the proposal will do through staff interviews.

Conclusion

The pilot's early result cannot answer leadership's question on its own, because it cannot separate texts from Saturday hours, seasons and a rebound from a bad winter. A controlled time series for the first six clinics and a randomized stepped wedge for the remaining eight use data Fairhaven already has and a rollout it already plans, while giving a far more trustworthy answer. Module Four will assemble the question, evidence and design into a first proposal.

References

Dimick, J. B., & Ryan, A. M. (2014). Methods for evaluating changes in health care policy: The difference-in-differences approach. JAMA, 312(22), 2401-2402. https://doi.org/10.1001/jama.2014.16153

Hemming, K., Haines, T. P., Chilton, P. J., Girling, A. J., & Lilford, R. J. (2015). The stepped wedge cluster randomised trial: Rationale, design, analysis, and reporting. BMJ, 350, Article h391. https://doi.org/10.1136/bmj.h391

Lopez Bernal, J., Cummins, S., & Gasparrini, A. (2017). Interrupted time series regression for the evaluation of public health interventions: A tutorial. International Journal of Epidemiology, 46(1), 348-355. https://doi.org/10.1093/ije/dyw098

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.

Skivington, K., Matthews, L., Simpson, S. A., Craig, P., Baird, J., Blazeby, J. M., Boyd, K. A., Craig, N., French, D. P., McIntosh, E., Petticrew, M., Rycroft-Malone, J., White, M., & Moore, L. (2021). A new framework for developing and evaluating complex interventions: Update of Medical Research Council guidance. BMJ, 374, Article n2061. https://doi.org/10.1136/bmj.n2061

Wagner, A. K., Soumerai, S. B., Zhang, F., & Ross-Degnan, D. (2002). Segmented regression analysis of interrupted time series studies in medication use research. Journal of Clinical Pharmacy and Therapeutics, 27(4), 299-309. https://doi.org/10.1046/j.1365-2710.2002.00430.x