The Dangers of Gathering Data: Surgeon-Specific Outcomes Revisited

 

 Buy Article Permissions and Reprints

ABSTRACT

The accuracy of risk adjustment is important in developing surgeon profiles. As surgeon profiles are obtained from observational, nonrandomized data, we hypothesized that selection bias exists in how patients are matched with surgeons and that this bias might influence surgeon profiles. We used the Society of Thoracic Surgeons risk model to calculate observed to expected (O/E) mortality ratios for each of six cardiac surgeons at a single institution. Propensity scores evaluated selection bias that might influence development of risk-adjusted mortality profiles. Six surgeons (four high and two low O/E ratios) performed 2298 coronary artery bypass grafting (CABG) operations over 4 years. Multivariate predictors of operative mortality included preoperative shock, advanced age, and renal dysfunction, but not the surgeon performing CABG. When patients were stratified into quartiles based on the propensity score for operative death, 83% of operative deaths (50 of 60) were in the highest risk quartile. There were significant differences in the number of high-risk patients operated upon by each surgeon. One surgeon had significantly more patients in the highest risk quartile and two surgeons had significantly less patients in the highest risk quartile (p < 0.05 by chi-square). Our results show that high-risk patients are preferentially shunted to certain surgeons, and away from others, for unexplained (and unmeasured) reasons. Subtle unmeasured factors undoubtedly influence how cardiac surgery patients are matched with surgeons. Problems may arise when applying national database benchmarks to local situations because of this unmeasured selection bias.

REFERENCES

• 1 Shahian D M, Normand S L, Torchiana D F et al.. Cardiac surgery report cards: comprehensive review and statistical critique.  Ann Thorac Surg. 2001;  72 (6) 2155-2168
  CrossrefPubMedGoogle Scholar
• 2 Grunkemeier G L, Zerr K J, Jin R. Cardiac surgery report cards: making the grade.  Ann Thorac Surg. 2001;  72 (6) 1845-1848
  CrossrefPubMedGoogle Scholar
• 3 Schneider E C, Epstein A M. Use of public performance reports: a survey of patients undergoing cardiac surgery.  JAMA. 1998;  279 (20) 1638-1642
  CrossrefPubMedGoogle Scholar
• 4 Marshall M N, Shekelle P G, Leatherman S, Brook R H. The public release of performance data: what do we expect to gain? A review of the evidence.  JAMA. 2000;  283 (14) 1866-1874
  CrossrefPubMedGoogle Scholar
• 5 Thomas J W, Hofer T P. Accuracy of risk-adjusted mortality rate as a measure of hospital quality of care.  Med Care. 1999;  37 (1) 83-92
  CrossrefPubMedGoogle Scholar
• 6 Naftel D C. Do different investigators sometimes produce different multivariable equations from the same data?.  J Thorac Cardiovasc Surg. 1994;  107 (6) 1528-1529
  PubMedGoogle Scholar
• 7 Edwards F H, Clark R E, Schwartz M. Coronary artery bypass grafting: the Society of Thoracic Surgeons National Database experience.  Ann Thorac Surg. 1994;  57 (1) 12-19
  CrossrefPubMedGoogle Scholar
• 8 Hattler B G, Madia C, Johnson C et al.. Risk stratification using the Society of Thoracic Surgeons Program.  Ann Thorac Surg. 1994;  58 (5) 1348-1352
  CrossrefPubMedGoogle Scholar
• 9 Edwards F H, Grover F L, Shroyer A L, Schwartz M, Bero J. The Society of Thoracic Surgeons National Cardiac Surgery Database: current risk assessment.  Ann Thorac Surg. 1997;  63 (3) 903-908
  CrossrefPubMedGoogle Scholar
• 10 Ghali W A, Ash A S, Hall R E, Moskowitz M A. Statewide quality improvement initiatives and mortality after cardiac surgery.  JAMA. 1997;  277 (5) 379-382
  CrossrefPubMedGoogle Scholar
• 11 Ivanov J, Tu J V, Naylor C D. Ready-made, recalibrated, or remodeled? Issues in the use of risk indexes for assessing mortality after coronary artery bypass graft surgery.  Circulation. 1999;  99 (16) 2098-2104
  CrossrefPubMedGoogle Scholar
• 12 Breiman L. Bagging predictors.  Mach Learn. 1996;  24 123-140
  PubMedGoogle Scholar
• 13 Blackstone E H. Breaking down barriers: helpful breakthrough statistical methods you need to understand better.  J Thorac Cardiovasc Surg. 2001;  122 (3) 430-439
  CrossrefPubMedGoogle Scholar
• 14 D'Agostino Jr R B. Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group.  Stat Med. 1998;  17 (19) 2265-2281
  CrossrefPubMedGoogle Scholar
• 15 Rosenbaum P R, Rubin D B. Reducing bias in observational studies using subclassification on the propensity score.  J Am Statist Assoc. 1984;  79 516-524
  CrossrefPubMedGoogle Scholar
• 16 Iezzoni L I. Risk Adjustment for Measuring Health Care Outcomes. 3rd ed. Chicago: Health Administration Press; 2003
  Google Scholar
• 17 Shahian D M, Torchiana D F, Normand S L. Implementation of a cardiac surgery report card: lessons from the Massachusetts experience.  Ann Thorac Surg. 2005;  80 (3) 1146-1150
  CrossrefPubMedGoogle Scholar
• 18 Goldstein H, Spiegelhalter D J. League tables and their limitations: statistical issues in comparisons of institutional performance (with discussion).  J R Stat Soc [Series A]. 1996;  159 385-443
  PubMedGoogle Scholar
• 19 Donabedian A, Bashshur R. An Introduction to Quality Assurance in Health Care. Oxford; New York: Oxford University Press; 2003
  Google Scholar
• 20 Kohn L T, Corrigan J, Donaldson M S. Institute of Medicine. (U.S.). Committee on Quality of Health Care in America. To Err is Human: Building a Safer Health System. Washington, D.C.: National Academy Press; 2000
  Google Scholar
• 21 Birkmeyer J D, Dimick J B, Birkmeyer N J. Measuring the quality of surgical care: structure, process, or outcomes?.  J Am Coll Surg. 2004;  198 (4) 626-632
  CrossrefPubMedGoogle Scholar
• 22 Shahian D M, Edwards F H, Ferraris V A Society of Thoracic Surgeons Quality Measurement Task Force et al.. Quality measurement in adult cardiac surgery: part 1—conceptual framework and measure selection.  Ann Thorac Surg. 2007;  83 (4, Suppl) S3-S12
  CrossrefPubMedGoogle Scholar
• 23 O'Connor G T, Plume S K, Olmstead E M The Northern New England Cardiovascular Disease Study Group et al.. A regional intervention to improve the hospital mortality associated with coronary artery bypass graft surgery.  JAMA. 1996;  275 (11) 841-846
  CrossrefPubMedGoogle Scholar
• 24 Grover F L, Johnson R R, Shroyer A L, Marshall G, Hammermeister K E. The veterans affairs continuous improvement in cardiac surgery study.  Ann Thorac Surg. 1994;  58 (6) 1845-1851
  CrossrefPubMedGoogle Scholar

Victor A. FerrarisM.D. Ph.D. 

Tyler Gill Professor, Division of Cardiovascular and Thoracic Surgery

A301 Kentucky Clinic, University of Kentucky, 740 South Limestone, Lexington, KY 40536-0284

Email: ferraris@earthlink.net