© The Author 2005. Published by Oxford University Press. All rights reserved. For Permissions, please e-mail: journals.permissions@oxfordjournals.org.
Brief Report |
An Automated Data Algorithm To Distinguish Screening and Diagnostic Colorectal Cancer Endoscopy Exams
Affiliations of authors: Kaiser Permanente Southern California, Research and Evaluation, CA (RH, VC, AMG); Southern California Permanente Medical Group, South Bay, CA (KRM)
Correspondence to: Reina Haque, PhD, MPH, Kaiser Permanente Southern California, Research and Evaluation, 100 S. Los Robles Ave., 2nd Floor, Pasadena, CA 91101 (e-mail: reina.haque{at}kp.org).
 |
ABSTRACT |
|---|
 TopNotes Abstract References |
|---|
Despite questions about accuracy, automated data are used increasingly for research and quality measurement. The goal of this study was to develop an automated data algorithm designed to distinguish screening and diagnostic endoscopy (sigmoidoscopy and colonoscopy) exams. We assessed the algorithm's ability to correctly classify the exams using paper medical records as the "gold standard." The algorithm used diagnostic codes to identify the indication of the endoscopies. The algorithm's ability to classify the indication varied by endoscopy exam. The sensitivities for identifying diagnostic sigmoidoscopy and colonoscopy were 48.1% and 23.8%, respectively. The algorithm missed most of the diagnostic endoscopies. Conversely, the sensitivities for identifying screening sigmoidoscopy and colonoscopy were high (87.9% and 84.4%, respectively) but were associated with low specificities. Our findings suggest that studies relying solely on automated data overestimate screening rates if indication is not considered. The automated algorithm presented here needs further improvements to better differentiate screening from diagnostic exams.
The Healthy People 2010 report declared increasing colorectal cancer screening a public health priority (1). Colorectal cancer screening exams include fecal occult blood test (FOBT), sigmoidoscopy, colonoscopy, and barium enema. These exams differ in cost, acceptability, risk, effectiveness, whether the exam allows visualization of the colon, and patient burden (2). Automated data provide opportunities to examine screening in large populations, but questions about accuracy and validity have not been addressed adequately. For example, health maintenance organizations use automated data to calculate quality measures despite the absence of indication information for the exams. Walter and colleagues recently demonstrated that colorectal cancer screening rates are overestimated in a Veteran's Administration medical center because the measures are driven by high rates of diagnostic testing (3). Ascertaining the indication of the exams may be accomplished only by detailed paper medical record review. Because this information does not exist in automated form, assessing the indication by medical record review is not feasible for large samples. The goal of this pilot study was to develop and evaluate an automated data algorithm designed to distinguish screening and diagnostic endoscopy (sigmoidoscopy and colonoscopy) exams. We assessed the algorithm's ability to correctly classify the exams using paper medical records as the "gold standard."
Kaiser Permanente Southern California (KPSC) cares for approximately 3 million members, of whom 13% are older than 50 years and targeted for colorectal cancer screening. Automated data tracks outpatient and inpatient care received. The study was approved by the KPSC Institutional Review Board.
Participants included all health plan members between the ages 50 and 70 years, who were continuously enrolled from 1998 to 2002, and completed an endoscopy during those years. Participants with a history of colorectal cancer were excluded (N = 1972). Endoscopies were identified using International Classification of Disease (ICD 9 CM) and Current Procedural Technology-4 codes (Fig. 1). In instances in which a participant completed multiple endoscopies in the 5-year period, we retrieved data for the first endoscopy. FOBT is the least expensive and is simplest to administer, but it has been shown to have poor sensitivity (4). Barium enemas cannot detect small lesions and are infrequently used due to cost (5). Therefore, these procedures were not included in the study.
|
The algorithm used automated data to presumptively classify the endoscopies as diagnostic or screening. Endoscopies were classified as diagnostic if automated data included certain gastrointestinal conditions in the year prior to the exam, or signs or symptoms or a FOBT in the 45 days prior. The study gastroenterologist (KRM) identified the conditions and signs and symptoms likely to result in diagnostic endoscopies (Fig. 1) (6). All other endoscopies were classified into the screening group.
We reviewed a stratified random sample of 220 medical records to assess the ability of the algorithm to correctly classify the endoscopies as screening or diagnostic. The 220 medical records were selected for review based on the algorithm's classification. Of 110 participants sampled for each type of endoscopy, the sample included 30 exams classified as diagnostic and 80 classified as screening by the algorithm. We selected as many participants as possible given our funding, and we oversampled among endoscopies classified as screening to generate the most information possible to inform future algorithm development.
Two trained abstractors reviewed medical records from 1997 to 2002 to confirm endoscopy use. The abstractors also assessed screening or diagnostic indications for the endoscopies, including the presence of gastrointestinal conditions or signs and symptoms. To minimize interrater variability, one abstractor reviewed all participants' medical records classified as a diagnostic exam while the second reviewed all participants' medical records classified as a screening exam. Abstractors classified the endoscopies as diagnostic if the exam was a follow-up to a previous abnormality or when clear-cut conditions or signs and symptoms were present, using the same list and time frames as the algorithm (Fig. 1). All other endoscopies were classified as screening.
We conducted cross-tabulations between the algorithm and medical review classification to examine the sensitivity, specificity, and 
. The classification after medical record review was considered the gold standard. Sensitivity indicates the probability that a diagnostic endoscopy was classified as such by medical records review. Specificity indicates the probability that a nondiagnostic endoscopy was classified as screening. The indicates the overall agreement between the two sources.
We identified more than 125 000 endoscopies performed during from 1998 to 2002 through the automated data. Of these, nearly 25% (31 322) were colonoscopies and the rest (93 993) were sigmoidoscopies. We excluded 32 of 220 endoscopies selected for the validation due to mismatches in participants' endoscopy dates. We examined the agreement between the automated algorithm and the medical record abstractions in the 188 remaining records.
The algorithm's ability to correctly classify the indication varied by endoscopy exam. The sensitivities for identifying diagnostic sigmoidoscopy and colonoscopy were 48.1% and 23.8%, respectively (Table 1). The algorithm missed most of the diagnostic endoscopies. Conversely, the sensitivities for identifying screening sigmoidoscopy and colonoscopy were high (87.9% and 84.4%, respectively) but were associated with low specificities. For sigmoidoscopy, the was 76.3%, but the agreement between the algorithm and the medical record review decreased to 44.2% for colonoscopy.
|
Of the 13 sigmoidoscopies classified as diagnostic by both the algorithm and the medical record review, most had a combination of conditions, but the most common condition recorded in the automated data was acute abdominal pain or rectal bleeding (n = 10). In addition to these conditions, record of a prior FOBT, personal history of colon polyps, or other nonspecified gastroenteritis and colitis in the automated data triggered a diagnostic classification for colonoscopies.
Our algorithm used existing gastrointestinal conditions or signs and symptoms codes to presumptively identify endoscopies that might be diagnostic rather than screening. Our results indicate that research and quality measures that rely solely on automated data may overestimate screening rates if exams are not classified as screening or diagnostic. We are unaware of previous studies that used an automated algorithm to differentiate screening from diagnostic colorectal cancer screening exams. Compared to medical records, the algorithm performed modestly in identifying screening endoscopies, but it performed poorly in identifying diagnostic endoscopies. The algorithm's ability to classify the indication for sigmoidoscopies ( = 76.3%) was better than for colonoscopies ( = 44.2%). For test of independence, the algorithm's performance for sigmoidoscopy was significant (P = .0002), but the association between the algorithm and medical chart review for colonoscopy exams may be due to chance (P = .356). The main limitation of the algorithm is that it missed the diagnostic colonoscopies; however, this procedure was performed on less than 25% of the members during the study period.
The allocation of the medical charts to abstractors for review based on the classification on the algorithm was unlikely to have created systematic bias for two main reasons. First, the abstractors were kept unaware of the classification of the algorithm. Second, despite the allocation in this manner, the two abstractors classified a similar percentage of the endoscopies as diagnostic (36% and 40%). The similar percentages also suggest that the preliminary algorithm did not adequately distinguish screening from diagnostic exams, especially for colonoscopies.
Previous studies of colorectal cancer screening use were limited by their ability to distinguish between tests that were completed for diagnostic versus screening purposes (2,3,7,8). An important strength of this study is that we examined colorectal cancer screening using automated data. Several of the past reports that examined colorectal cancer screening rates were either based on self-reports or did not attempt to exclude symptomatic individuals (7,9–11). In such studies, and in quality measure calculations, colorectal cancer screening utilization may be overestimated (12–15).
In a previous pilot study, we examined the validity of the automated data systems to identify all endoscopies that were performed. Automated endoscopy data was compared with those recorded in the medical records (the gold standard) for 50 members. The sensitivity comparing the automated data with the medical records was high (90%), and specificity was 92%. The positive predictive value was also high (90%). Therefore, most of the endoscopy procedures are captured in the automated data available in this health plan.
It is also unknown if other automated health care data systems would obtain similar results in distinguishing diagnostic and screening endoscopy exams when applying this algorithm. However, our preliminary algorithm identifies a provisional list of gastrointestinal conditions and signs and symptoms conditions that should be improved and tested in other settings to help distinguish diagnostic from screening exams.
Given the low sensitivities and specificities of the preliminary algorithm, improved coding of indication would enhance the ability to distinguish screening from diagnostic endoscopies. Such a task would require organizational system changes but would result in improved research and measures of health care quality. The present algorithm classified the endoscopies as diagnostic based on a limited number of gastrointestinal conditions and signs and symptoms, and it could be enhanced by expanding this provisional list or determining if certain preventive health measures (such as mammograms in women or prostate antigen screening among men) occurred around the time of endoscopy.
|
NOTES |
|---|
This study was conducted within the Cancer Research Network, a consortium of research organizations affiliated with nonprofit integrated healthcare delivery systems and the National Cancer Institute. The Cancer Research Network (CRN) consists of the research programs, enrollee populations, and databases of 11 integrated healthcare organizations that are members of the HMO Research Network. The overall goal of the CRN is to increase the effectiveness of preventive, curative, and supportive interventions that span the natural history of major cancers among diverse populations and health systems, through a program of collaborative research.
Supported by grant U19 CA 79689 from the National Cancer Institute, Increasing Effectiveness of Cancer Control Interventions, Edward H. Wagner, principal investigator.
|
REFERENCES |
|---|
|
TopNotesAbstractReferences |
|---|
(1) Healthy People 2010. Available at: http://www.healthypeople.gov/document/HTML/Volume1/03Cancer.htm [last accessed: December 13, 2004].
(2) Agency for Health Care Policy and Research. Colorectal cancer screening. Summary, evidence report: number 1. Available at: http//www.ahrq.gov/clinic/colorsum.htm [last accessed: November 1, 2002].
(3) Walter LC, Davidowitz NP, Heineken PA, Covinsky KE. Pitfalls of converting practice guidelines into quality measures: lessons learned from a VA performance measure. JAMA 2004;291:2466–70.
(4) Ransohoff DF, Lang CA. Screening for colorectal cancer with the fecal occult blood test: a background paper. American College of Physicians. Ann Intern Med 1997;126:811–22.
(5) Pignone M, Rich M, Teutsch SM, Berg AO, Lohr K. Screening for colorectal cancer in adults at average risk: A summary of the evidence for the US Preventive Services Task Force. Ann Intern Med 2002;137:I138.
(6) Walsh JM, Posner SF, Perez-Stable E. Colon cancer screening in the ambulatory setting. Prev Med 2002;35:209–18.[CrossRef][ISI][Medline]
(7) Breen N, Wagener DK, Brown ML, Davis WW, Ballard-Barbash R. Progress in cancer screening over a decade: results of cancer screening from the 1987, 1992, and 1998 National Health Interview Surveys. J Natl Cancer Inst 2001;93:1704–13.
(8) Freeman JL, Klabunde CN, Schussler N, Warren JL, Virnig BA, Cooper GS. Measuring breast, colorectal and prostate cancer screening with Medicare claims data. Med Care 2002;40:IV36–42.
(9) Erban S, Zapka J, Puleo E, Vickers-Lahti, M. Colorectal cancer screening in Massachusetts: measuring compliance with current guidelines. Eff Clin Pract 2001;4:10–7.[Medline]
(10) Zapka JG, Puleo E, Vickers-Lahti M, Luckmann R. Healthcare system factors and colorectal cancer screening. Am J Prev Med 2002;23:28–35.[CrossRef][ISI][Medline]
(11) McMahon LE, Wolfe RA, Huang S, Tedeschi P, Manning W, Edlund MJ. Racial and gender variation in use of diagnostic colonic procedures in the Michigan Medicare population. Med Care 1999;37:712–7.[CrossRef][ISI][Medline]
(12) Mandelson MT, LaCroix AZ, Anderson LA, Nadel MR, Lee NC. Comparison of self-reported fecal occult blood testing with automated laboratory records among older women in a health maintenance organization. Am J Epidemiol 1999;150:617–21.
(13) Mandelson MT, Curry SJ, Anderson LA, Nadel MR, Lee NC, Rutter CM, et al. Colorectal cancer screening participation by older women. Am J Prev Med 2000;19:149–54.[CrossRef][ISI][Medline]
(14) Lipkus IM, Rimer BK, Lyna PR, Pradhan AA, Conaway M, Woods-Powell CT. Colorectal screening patterns and perceptions of risk among African-American users of a community health center. J Community Health 1996;21:409–27.[CrossRef][ISI][Medline]
(15) Hiatt RA, Perez-Stable EJ, Quesenberry C Jr, Sabogal F, Otero-Sabogal R, McPhee SJ. Agreement between self-reported early cancer detection practices and medical audits among Hispanic and non-Hispanic white health plan members in northern California. Prev Med 1995;24:278–85.[CrossRef][ISI][Medline]
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||