ARCHIVED - Chronic Diseases in Canada

Volume 30, no. 4, September 2010

Myocardial infarction and the validation of physician billing and hospitalization data using electronic medical records

K. Tu, MD (1,2,3); T. Mitiku, MSc (1); H. Guo, MSc (1); D. S. Lee, MD, PhD (1,4); J.V. Tu, MD, PhD (1,5)

https://doi.org/10.24095/hpcdp.30.4.06

Author References

1. Institute for Clinical Evaluative Sciences (ICES), Toronto, Canada
2. Department of Family and Community Medicine, University of Toronto, Toronto, Canada
3. Toronto Western Hospital Family Health Team, University Health Network, Toronto, Canada
4. Department of Medicine, University Health Network, University of Toronto, Toronto, Canada
5. Department of Medicine, Sunnybrook Health Sciences Centre, University of Toronto, Toronto, Canada

Correspondence: Karen Tu, c/o (ICES), G1 06, 2075 Bayview Avenue, Toronto, Ontario M4N 3M5; Tel.: (416) 480-4055 x 3871; Fax: (416) 480-6048; Email: karen.tu@ices.on.ca.

Abstract

Objective: Population-based identification of patients with a myocardial infarction is limited to patients presenting to hospital with an acute event. We set out to determine if adding physician billing data to hospital discharge data would result in an accurate capture of patients who have had a myocardial infarction.

Methods: We performed a retrospective chart abstraction of 969 randomly selected adult patients using data abstracted from primary care physicians on an electronic medical record in Ontario, Canada, as the reference standard.

Results: An algorithm of 3 physician billings in a one-year period with at least one being by a specialist or within a hospital or emergency room plus one hospital discharge abstract performed with a sensitivity of 80.4% (95% CI: 69.5-91.3), specificity of 98.0% (95% CI: 97.1-98.9), positive predictive value of 69.5% (95% CI: 57.7-81.2), negative predictive value of 98.9% (95% CI: 98.2% to 99.6%) and kappa statistic of 0.73 (95%
CI: 0.63-0.83).

Conclusion: Using a combination of hospital discharge abstracts and physician billing data may be the best way of assessing trends of MI occurrence over time since it increases the capture of MI beyond those patients who have been hospitalized.

Keywords: myocardial infarction, medical care costs, electronic health records, patient discharge, validation studies, Ontario

Introduction

The validity of using hospital claims data to identify myocardial infarction (MI) patients compared to hospital records varies depending on where the validation took place, what hospital code was assessed and how the patient population was selected.^1-5a Similarly, the validity of hospital claims data compared to registry data varies widely.^6-11 Regardless, estimates indicate that at least a quarter of all MIs are silent or unrecognized.^12-15 Some patients have MIs out of country or province that, as a result, are not captured in a provincial hospital discharge abstract database. Since the prognosis of silent or unrecognized MIs is worse than that of recognized MIs of similar severity, this has very important clinical implications.^14b,16a,17a Previous administrative data validation studies have neither looked at using outpatient clinic data in a primary care or family practice setting nor assessed the use of outpatient physician billing claims data to identify patients with MIs. As a result, when we report on the occurrence of MIs within a province or across the country, we are restricted to methods for reporting acute MIs that present to hospital or that are in mortality records.

All provinces in Canada keep a record of both hospitalization data and physician billing data. However, physician billing data only require recording of one diagnostic code with each patient visit despite that patients often present with multiple reasons for a visit and often have multiple medical conditions. Vali­dated algorithms using both physician billing data and hospital discharge abstracts have been successfully used to identify patients with chronic disease conditions who do not necessarily require hospitalization, such as hypertension^18a and diabetes.^19a

Using data from pridetermine if the admary care physician electronic medical records (EMRs) as a reference standard, we set out to dition of physician billing data to hospital discharge abstracts could accurately identify patients who have had an MI, which could then be applied to determine the occurrence of MI at a population level.

Methods

We collected data from 17 physicians who had been using Practice Solutions® EMR for at least 2 years; this ensured that their EMRs were populated with a full practice of patients. Data from the EMR were extracted from June to December of 2007, de-identified, encrypted and then transferred electronically to the Institute for Clinical Evaluative Sciences ICES in a secure fashion. ICES is a prescribed entity under the Personal Health Information Protection Act,20 and the data were handled as per ICES’s standard operating pro­cedures to preserve patient privacy and confidentiality.21

Our convenience sample of 17 physicians was 29.4% female, 41.7% in rural practice, and all but one in a group practice. The average number of years in practice was 20.5 years (standard deviation [SD] = 10.2) and the average length of time using the EMR was 7.4 years (SD = 7.3). The average age of our patient cohort was 49.0 years (SD = 17.2) and 53.7% were female.

We took a 5% random sample of the EMR patients aged 20 years and over as of Dec 31, 2006, with valid health card numbers and with at least two progress note entries in the three years prior to the date the data were downloaded. Three trained chart abstractors reviewed all of the 969 patient records in this random sample up to March 31, 2007, and each entry was scored to indicate a definite MI, a possible MI, a family history of MI, or not an MI. A definite MI was defined as having an MI recorded in the cumulative patient profile, in a progress note, on an electrocardiogram (ECG), in a diagnostic test or procedure, or in a hospital discharge summary, emergency record, consultation letter or operative note. The resulting abstraction classification for each patient was tabulated; patients were classified as having had an MI if there were one or more entries denoting definite MI and as a possible MI if there were only entries scored as possible MI and no entries scored as definite MI. Intra-observer reliability testing resulted in kappa statistic values exceeding 0.80, indicating very good agreement; inter-observer reliability testing resulted in kappa statistic values exceeding 0.85 for all comparisons between the three abstractors, also indicating very good agreement.

Next, all the MIs that were classified as definite or possible were reviewed by a physician expert on the research team and classified according to the level of evidence for MI: 1) solid MI evidence—a hospital discharge summary, or a consultation letter describing an MI or listing an MI in the past medical history, or a catheterization report describing complete occlusion of a coronary artery; 2) MIs that were only recorded in the cumulative patient profile entered in the family physician office; 3) weaker MI evidence—MIs that were described in a diagnostic test such as an echocardiogram (“wall motion abnormality in keeping with infarct”) or a sestamibi cardiac scan, but not interpreted as depicting an MI as reported in a consultation letter; and 4) ECG evidence of an MI with no other supporting documentation. The aim of this classification was to assess the quality of MI evidence in the EMR; sensitivity analysis was performed using differing levels of evidence for MI as the reference standard.

Patient health card numbers were replaced with a unique identification number and anonymously linked to the Ontario administrative data holdings housed at ICES. The data holdings include the Canadian Institute for Health Information hospital discharge abstract database (CIHI DAD) that records the diagnosis most responsible for a hospital admission and up to 15 comorbid conditions using the International Classification of Diseases and Related Health Problems, 9th and 10th Revisions (ICD-9 codes 410 and 412 prior to fiscal year 2002 and ICD-10 codes I21, I22, I25.2 after fiscal year 2002). We used the most responsible diagnostic codes or any of the secondary diagnostic codes to indicate that a patient had had an MI, and CIHI data from 1988 until fiscal 2006.

The random sample of 969 patients became the reference standard against which we compared various case definitions using combinations of data from CIHI, the Ontario Health Insurance Plan (OHIP), the National Ambulatory Care Reporting System (NACRS) and the Same Day Surgery (SDS) databases. OHIP physician billing claims data records over 95% of the outpatient visits for the residents of Ontario,22 as well as the type of physician billing a claim (e.g. type 00 = general practitioner or family physician) and the location of the encounter (i.e. hospital, emergency room, out-patient clinic, nursing home); OHIP data are available from mid-1991 until the most recent quarter. NACRS is a data collection tool that is used to capture information on patient visits to hospital and community-based ambulatory care, outpatient clinics and emergency departments from July 2000 to March 2007. NACRS also includes same day surgery procedures as of April 2003; prior to this the SDS database captured this information.

We also assessed whether restricting the physician billing claim to only those billed by a specialist or by a general practitioner/family physician in a hospital or emergency room affected the results.

We calculated the sensitivity of the administrative data as the proportion of MI patients identified according to the reference standard of manual EMR abstraction compared to those identified by the administrative algorithm. Specificity was calculated in the same manner except that it was based on individuals without an MI. Positive predictive value (PPV) was defined as the proportion of MI patients identified by the administrative algorithm that was confirmed by the reference standard, and negative predictive value (NPV) was defined similarly for patients who did not have an MI according to the manual EMR abstraction. Since both data sources have limitations, we calculated kappa between the EMR data and the administrative data. All proportions were calculated with 95% confidence intervals (CI) using the binomial approximation method in SAS version 9.1.^* All analyses were conducted using SAS version 9.1.

This study received ethics approval from the Sunnybrook Health Sciences Centre Research Ethics Board.

Results

Of the 969 patients included in our study, 51 had solid evidence of an MI documented in the EMR and 918 did not. On examining the strength of the evidence for MI, excluding patients with only ECG evidence or weaker evidence resulted in the highest sensitivity (60.8%, 95% CI: 47.4-74.2) with the least sacrifice of PPV (88.6%, 95% CI: 78.0-99.1) (see Table 1). Accordingly, we adopted this level of EMR evidence to identify patients who have had an MI to compare with the accuracy of other administrative data sources to identify such patients (see Table 2). Generally, we found that using hospital discharge abstracts alone to identify patients who have had an MI was not sensitive enough and likely underestimates the true population occurrence of MIs. For instance, only 31 of the 51 patients in our reference group who had an
MI had a discharge abstract that mentioned the MI.

 

Abbreviations: CI, confidence interval; ECG, electrocardiogram; EMR, electronic medical record; MI, myocardial infarction; N, sample size; NPV, negative predictive value; PPV, positive predictive value.

Note: N = 969 (reference sample).

 

Abbreviations: ECG, electrocardiogram; EMR, electronic medical record; MI, myocardial infarction; N, sample size; NPV, negative predictive value; PPV, positive predictive value.

Note: N = 969 (reference sample).

 

The results of accuracy tests using secondary administrative data sources plus hospital discharge data are presented in Table 3. The small gain in sensitivity with the addition of
NACRS or SDS (62.7%, 95% CI: 49.5-76.0) was reflected in a similar drop in PPV (84.2%, 95% CI: 72.6-95.8), indicating that using these administrative databases did not contribute to the accuracy of the calculations.

 

Abbreviations: CI, confidence interval; MI, myocardial infarction; NACRS, National Ambulatory Care Reporting System; NPV, negative predictive value; SDS, Same Day Surgery; PPV, positive predictive value.

^a Evidence for MI as documented in a cardiac specialist note or hospital discharge summary, or recorded in the cumulative patient profile (prevalence = 5.3%) as the reference standard.

 

Adding a single physician billing claim to the hospital discharge abstract substantially increased the sensitivity (86.3%, 95% CI: 76.8-95.7) but decreased the PPV to an unacceptably low rate (41.5%, 95% CI: 32.1-50.9). Requiring two physician billing claims for MI increased the sensitivity by 20 percentile points (82.4%, 95% CI: 71.9-92.8) but dropped the PPV by more than 25 percentile points (60.9%, 95% CI: 49.4-72.4) compared to using the hospital discharge abstract alone. Adding an additional year for the second MI physician billing claim to meet the case definition did not improve sensitivity (82.4%, 95% CI: 71.9-92.8) and slightly decreased PPV (59.2%, 95% CI: 47.7-70.6) compared to using 2 physician billing claims within one year. Requiring a third physician billing claim dropped the sensitivity by 2 percentile points (80.4%, 95% CI: 69.5-91.3) but increased the PPV by 6 percentile points (66.1%; 95% CI: 54.3-77.9) compared to using only 2 physician billing claims in a one-year period or a hospital discharge abstract (see Table 4).

 

Abbreviations: CI, confidence interval; MI, myocardial infarction; NPV, negative predictive value; PPV, positive predictive value.

^a Evidence for MI as documented in a cardiac-related specialist note or hospital discharge summary, or recorded in the cumulative patient profile (prevalence = 5.3%) as the reference standard.

 

Requiring the physician billing claims to be from a specialist or a general practitioner/family physician with an encounter in a hospital or an emergency room slightly dropped the sensitivity (84.3%, 95% CI: 74.3-94.3) but increased the PPV slightly more (48.9%, 95% CI: 38.4-59.3) compared to using physician billing claims without regard to location of encounter or type of physician. Only requiring one of the physician billing claims to meet this criteria increased the PPV slightly more (65.6%, 95% CI: 54.0-77.3) with less of a sacrifice of sensitivity (82.4%, 95%
CI: 71.9-92.8). Three physician billing claims in one year with at least one of the claims from an encounter in a hospital or an emergency room or by a specialist resulted in the largest increase in sensitivity (80.4%, 95% CI: 69.5-91.3) combined with the least sacrifice of specificity (98.0%, 95% CI: 97.1-98.9) and PPV (69.5%, 95% CI: 57.7-81.2) compared to using a hospital discharge abstract alone to identify patients who have had an MI (see Table 4).

Discussion

Similar to other chronic disease conditions such as diabetes19b and hypertension,18b this validation study shows that physician billing data help identify patients who have had a silent or unrecognized MI or an MI not recorded in the hospital discharge data that would not otherwise be captured. Given the detrimental effect of unrecognized MIs on long-term mortality and major cardiac events,14c,16b,17b knowing the occurrence of these events in a population has important implications for health service planning and for assessment of secondary treatment and prevention strategies and quality measurement. Our findings also show that the addition of emergency room records does not improve the accuracy of identifying patients with an MI; thus provinces without such records would not be at a disadvantage in identifying MI patients through their administrative data.

Administrative data have the potential to facilitate the assessment of MI occurrence in a population. Note that the PPV in the administrative data algorithms using physician billing claims may not be high enough to avoid the accumulation of false positives and therefore these may overestimate the true occurrence of MI in a population. However, given that the sensitivity of these algorithms is in the 80% range—suggesting that not all cases are picked up—the possible over-reporting may not be that high. Further, administrative data are confined to patients who have come in contact with the medical system; it is possible that there are patients in the general population who have had an unrecognized MI but who have never sought medical attention or had their MI diagnosed by a physician.

It is difficult to compare our findings with previous validation studies as they used a reference standard derived from either hospital charts or clinical patient registries based upon hospitalization, and none have looked at using physician billing claims to identify MIs that did not result in hospitalization. Another Ontario-based validation study looked for ICD-10 code I21 for acute MI as the most responsible and secondary diagnoses and found a sensitivity of 89% and PPV of 87% for the most responsible diagnosis, but a sensitivity of 78% and PPV of 76% when using the hospital chart as the reference standard. These results are higher than ours since the data sources were restricted to patients who were hospitalized for their MI.1 Our results, which used physician claims data plus hospital discharge abstracts compared to family physician charts, performed similarly to a study in Australia that used registry data as the reference standard and showed a sensitivity of 79% and PPV of 66% for acute MI using hospital diagnostic codes.^10b Our study also had similar results to one in the US that compared hospital coding to hospital discharge records as a reference standard with a sensitivity of 81% and a PPV of 55%.^5b

Limitations

We only used data from a convenience sample of physicians who were on one particular EMR. We do not know if physicians on this particular EMR code differently to physicians who are not on an EMR or are on a different one. However, it is unlikely that the patient population or disease occurrence is different for physicians who are on an EMR compared to those who are not.

Second, it is possible that some of the weaker evidence for MI or the ECG-only evidence may, with time, turn out to be real or silent MIs; capture of MIs in the EMR may depend on the length of time the patient has been on the EMR, how well the physician captured historic patient medical information in the EMR, or when the abnormalities were captured in relation to the date the EMR data were downloaded.

Third, we were unable to fully apply WHO criteria for MI as only the interpretation of ECG wave forms are typically captured in the EMR; troponin values are also generally not captured in the EMR. Fourth, data were only available from CIHI as of 1988 and from OHIP as of 1991. Thus patients who had an MI before then may not be captured in the administrative data.

Last, our analysis did not take into account the 52.1 per 100 000 people who die of an MI on the way to the hospital23 as our data were confined to patients registered on the physician EMR; patients who died may have been taken out of the physician EMR.

Nonetheless, data contained within an EMR appear to be rich and reasonably comprehensive and can be used to validate case ascertainment algorithms using administrative data. Using a combination of hospital discharge abstracts and physician billing data increases the capture of MI beyond patients who have been hospitalized and may best be used for assessing trends of MI occurrence over time.

Acknowledgements

This work was funded by a Canadian Institutes of Health Research Team grant in Cardiovascular Outcomes Research to the Canadian Cardiovascular Outcomes Research Team (CCORT) and a grant from the Public Health Agency of Canada.

This study was supported by the Institute for Clinical Evaluative Sciences ICES, which is funded by an annual grant from the Ontario Ministry of Health and Long-Term Care (MOHLTC). The opinions, results and conclusions reported in this paper are those of the authors and are independent from the funding sources. No endorsement by ICES or the Ontario MOHLTC is intended or should be inferred.

Dr. Douglas Lee is a Clinician Scientist of the Canadian Institutes of Health Research. Dr. Jack Tu is supported by a Canada Research Chair in Health Services Research and a Career Investigator award from the Heart and Stroke Foundation of Ontario.

*