HIM 675 Module 6 Instrument Short Paper Example

Same Chart, Same Answer? Designing and Testing the Abstraction Tool Used in the Tulsa Flag Study

[Student Name]

Southern New Hampshire University

HIM 675: Research Methods and Evaluation

Module Six Short Paper

[Instructor Name]

[Date]

The organization, setting and figures below are a composite written as a model document. No real employer, client, colleague or patient is described.

Same Chart, Same Answer? Designing and Testing the Abstraction Tool Used in the Tulsa Flag Study

A study of whether present-on-admission flags are accurate depends on a gold standard that is itself trustworthy. At Cimarron Heights Medical Center, the composite Tulsa teaching hospital where the study will take place, the gold standard is a blind re-abstraction of each record by a credentialed coder. If two abstractors reading the same chart reach different conclusions, the study's results will reflect abstractor disagreement as much as coding error. This paper describes the abstraction tool, how its content was checked and how its reliability was tested in two pilot rounds before data collection begins.

The Abstraction Tool

The tool is an electronic form with four sections. The first records identifiers: a study number in place of the medical record number, the condition under review and the admission source. The second records the evidence at admission, including whether a structured skin and risk assessment was documented, its date and time, any skin finding, the Braden score and any documentation of the condition in the emergency department, transfer records or physician history within 24 hours. The third records the abstractor's flag, using the same five values as the claim, with a free-text field explaining the evidence relied on. The fourth records process variables: whether the coder queried a clinician and the query's outcome.

Each condition has written decision rules. For pressure injuries, a flag of yes requires documentation of a wound at the same site and of the same or lower stage within 24 hours of admission, using the stage definitions in the revised national staging system (Edsberg et al., 2016). A deep tissue injury first documented on day two or three is treated as clinically undetermined unless a physician states it was present on arrival. For catheter infections, yes requires that the catheter and the infection both preceded admission. For blood clots and fall injuries, the rules rely on imaging dates and incident reports. Abstractors are blinded to the coded flag, following the approach of Goldman et al. (2011).

Content Validity

A tool is valid if it measures what it claims to measure. Before testing reliability, three experts reviewed every field and rule: a certified wound care nurse, the hospital's coding quality lead and a clinical documentation specialist. They rated each rule as essential, useful or unnecessary and suggested changes. Two changes resulted. The wound nurse added the rule about deep tissue injuries, because those injuries often appear after admission even though the damage began before it, and the coding lead asked for a field recording scanned outside records, which coders sometimes miss. The tool also maps to the data quality dimensions described by Weiskopf and Weng (2013), particularly completeness of admission documentation and concordance between the claim and the clinical record, which supports its fit with the study's framework.

Pilot Round One

Two credentialed coders, neither of whom coded the original records, independently abstracted 30 records chosen to include all four conditions. Table 1 shows their flags for the 30 records.

Table 1. Pilot Round One Agreement Between Abstractors (n = 30)

Note. Records with insufficient documentation or clinically undetermined values were analyzed separately; none occurred in round one.

Why Kappa, Not Agreement

The abstractors agreed on 27 of 30 records, or 90%, which sounds excellent. But some agreement would happen by chance. Cohen's kappa adjusts for that. Here, chance agreement is about 50%, calculated from how often each abstractor said yes and no, and kappa equals 0.90 minus 0.50, divided by 1 minus 0.50, or approximately 0.80. McHugh (2012) cautions that raw percent agreement overstates reliability and proposes interpreting kappa from 0.60 to 0.79 as moderate, 0.80 to 0.90 as strong and above 0.90 as almost perfect, noting that health research data often need at least strong agreement. A kappa at the boundary of moderate and strong was not good enough for a gold standard.

The three disagreements were examined together. Two involved deep tissue injuries documented on the second day, and one involved a transfer record scanned into the chart days after admission. All three traced to rules that were unclear, not to careless reading.

Revisions and Round Two

Three rules were rewritten. The deep tissue injury rule now states exactly which documentation counts as a physician statement. The scanned records rule now says that outside records count only if their content describes the patient's condition before arrival, regardless of when they were scanned. A third rule clarified how to treat a pressure injury documented at a different site from the coded one. The abstractors then reviewed 20 new records. They agreed on 19, and kappa rose to about 0.89, within the strong range. The study will proceed with this version of the tool, and a 10% sample of study records will be double-abstracted throughout data collection to confirm that reliability holds.

Training and Drift

Reliability can fade once a study is under way, a problem often called drift. The two abstractors spent two hours in training ahead of the pilot, working through ten practice records with the decision rules and discussing each answer, and they will repeat a short refresher with five new practice records after every hundred study records. Disagreements in the ongoing double-abstraction sample will be resolved by a third reviewer, the coding quality lead, and logged with the rule involved, so that any rule producing repeated disagreement can be clarified and the change dated in the study record. If kappa in any batch falls below 0.80, abstraction will pause until the cause is found. These steps cost a few hours but protect the credibility of every number the study reports. The full set of rules, with the date of each revision, will be attached to the proposal as an appendix so that the committee can judge them directly.

Conclusion

The abstraction tool now has written rules for every condition, expert review behind its content and strong agreement between abstractors after one round of revision. The pilot also taught a lesson about measurement: 90% agreement looked reassuring until chance agreement was removed. The same caution will apply when the study compares coded flags with the gold standard.

References

Edsberg, L. E., Black, J. M., Goldberg, M., McNichol, L., Moore, L., & Sieggreen, M. (2016). Revised National Pressure Ulcer Advisory Panel pressure injury staging system. Journal of Wound, Ostomy and Continence Nursing, 43(6), 585-597. https://doi.org/10.1097/WON.0000000000000281

Goldman, L. E., Chu, P. W., Osmond, D., & Bindman, A. (2011). The accuracy of present-on-admission reporting in administrative data. Health Services Research, 46(6pt1), 1946-1962. https://doi.org/10.1111/j.1475-6773.2011.01300.x

McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276-282. https://doi.org/10.11613/BM.2012.031

Weiskopf, N. G., & Weng, C. (2013). Methods and dimensions of electronic health record data quality assessment: Enabling reuse for clinical research. Journal of the American Medical Informatics Association, 20(1), 144-151. https://doi.org/10.1136/amiajnl-2011-000681